3MSP2: Sequential picking planning for multi-fruit congregated tomato harvesting in multi-clusters environment based on multi-views

•Sequential picking planning method 3MSP2 is proposed for clusters of congregated tomatoes.•The eye-on-shoulder and eye-in-hand two RGB-D camera setting are preset in robotic piking.•Optimal sequence planning method is presented for tomato clusters division and picking order.•Optimal Sequential Grasping posture estimation method is proposed for efficient and low-damage.•Greenhouse picking validates average cycle time 8.6 s, collision-free picking success rate 70.9 %. Fast and effective picking planning from candidate fruits is important to improve the efficiency and success rate of robotic selective harvesting. However, it is a challenge for tomato picking decisions to simultaneously meet low- damage and high-performance requirements in multi-fruits congregation and multi-clusters environments. For such robotic application scenario, a novel sequential picking planning method is proposed for tomatoes harvesting sequence and picking posture estimation combined eye-on-shoulder and eye-in-hand views. First, the detection and localization of multiple tomatoes in both views are achieved by Yolov5. Then, a cluster Optimal Sequence Planning method was developed by us based on the view of eye-on-shoulder. The method combined the density clustering idea and the shortest movement path to automatically classify multiple neighboring tomatoes into candidate picking clusters and make decisions on the optimal picking order of the clusters. Finally, we developed a tomato Optimal Sequential Grasping method based on the view of eye-in-hand. The method combines collision-free constraints and a shortest time objective to determine the optimal grasping posture estimate and the optimal post-grasping observation position estimate for the target tomato. In indoor picking experiments, the proposed method improves the collision-free picking success rate of picking by 17.2 % and 36.1 %, and the picking efficiency by 25 % and 41.4 %, respectively, compared with the picking planning methods based on random traversal and scanning traversal. In the picking experiments in industrial tomato greenhouse, the collision-free picking success rate reaches 70.9 %, and the average cycle time for each picking task from target-recognizing to success-picking is 8.6 s. This approach offers a promising solution for picking planning of harvesting robot to lower damages and improve success rate, efficiency in the congregated fruit environment.