Integrated detection of citrus fruits and branches using a convolutional neural network

Integrated Detection of Fruits and Branches. [Display omitted] •Detailed classification principle of orchard citrus tree for robotic harvesting.•Discrete marking and detection method for random growth and irregular shapes branches.•A multiple parameters constraint algorithm for branch reconstruction.•Location of citrus fruits in different growth status and surrounding branches. The key technology for a fruit picking robot is to identify fruits in different occlusion states. Based on the mask regional convolutional neural network (Mask R-CNN) and a branch segment merging algorithm, an integrated system was developed to simultaneously detect and measure citrus fruits and branches. A training dataset was constructed for fruit and tree appearance, including single fruit, multiple fruits, occluded fruits, branches and trunk. A segmental labeling method for random and irregular branches is proposed to improve the precision of the Mask R-CNN. Based on the segmental mask regions identified by this model, a more precise bounding box is obtained by calculating the minimum enclosing rectangle of mask regions. Then, a branch segment merging algorithm reconstructs branches and the trunk. Diameters of fruits and branches are obtained by mapping the color image onto the depth image. The average precision of fruit and branch recognition are 88.15% and 96.27%, respectively. The average measurement error of fruits’ transverse diameters, fruits’ longitudinal diameters, and branch diameters are 2.52, 2.29, and 1.17 mm, respectively. Experiments show the detection system has good performance for all types of fruits and occlusions. This vision system can effectively help the robot to plan the appropriate picking path and avoid obstacles.