NDMFCS: An automatic fruit counting system in modern apple orchard using abatement of abnormal fruit detection

•Abatement of abnormal fruit detection led to more accurate fruit detection.•Tracked trunk offered a reliable reference displacement in consecutive video frames.•Identity document (ID) assignment efficiently counted fruits in modern apple orchards.•Confidence threshold alleviated over count caused by fruits on the ground.•Distance threshold alleviated ID switch caused by fruits in the back row of trees. Automatic fruit counting is an important task for growers to estimate yield and manage orchards. Although many deep-learning-based fruit detection algorithms have been developed to improve performance of automatic fruit counting systems, abnormal fruit detection has often been caused by these algorithms detecting non-target fruits that have similar growth characteristics to target fruits. For abnormal fruit detection, detected fruits in the back row of the tree were defined as DFBRT, while detected fruits on the ground were defined as DFG. Both of them would result in a higher number of fruits counting than the ground truth. This study proposes an automatic fruit counting system called NDMFCS (Normal Detection Matched Fruit Counting System) to solve this problem for improving fruit counting accuracy in modern apple orchard. NDMFCS consists of three sub-systems, i.e. object detection based on You Only Look Once Version 4-tiny (YOLOv4-tiny), abatement of abnormal fruit detection based on threshold, and fruit counting based on trunk tracking and identity document (ID) assignment. YOLOv4-tiny was selected to implement detection of fruits and trunks, whose output is confidence and pixel coordinates of detected object. The DFBRT and DFG were abated by thresholds to improve detection performance of fruit. This meant that detected fruits were removed when their distance from camera is further than a distance threshold or the confidence of fruit detection is less than a confidence threshold. Finally, fruit counting was implemented by trunk tracking and ID assignment, where each fruit was assigned a unique tracking ID. Results on 10 sets of original videos indicated that average fruit detection precision was improved from 89.1% to 93.3% after abatement of abnormal fruit detection. Also, Multiple Object Tracking Accuracy and Multiple Object Tracking Precision were improved on average by 4.2% and 3.3%, respectively, while average ID Switch Rate was decreased on average by 1.1%. And average fruit counting accuracy was improved to 95.0% by 4.2%. Coefficient of determinat