An improved cascade R-CNN and RGB-D camera-based method for dynamic cotton top bud recognition and localization in the field

•According to the needs of cotton top cutting in the field, the idea of identifying the single cotton top bud by computer vision technology and removing it by mechanical arm is proposed for the first time. We also proposed an improved Cascade R-CNN model to identify cotton top buds in the field in different time of the day with high-precision and rapid.•A three-dimensional position compensation method of cotton top bud under dynamic conditions is proposed. According to the platform moving speed and time consumption of target recognition, the position compensation is carried out to realize the accurate recognition and positioning of cotton top buds with different moving speeds. It provides guarantee for accurate top cutting operation of cotton during movement.•The method proposed in this manuscript can be applied to the dynamic operation of agricultural robots, and provide ideas for the realization such as precise spraying and precise weeding. Cotton top cutting is an indispensable part of cotton planting. Cotton top bud detection and localization are highly challenging tasks because of tiny targets, dense growth and varying illumination. To achieve the automatic cutting of cotton buds in the field, a cotton top bud recognition and location algorithm adapted to a moving platform in the field based on images acquired by a red, green, blue and depth (RGB-D) camera was developed. In this study, an improved Cascade R-CNN network was proposed to detect cotton top bud regions on RGB images, and three-dimensional (3D) coordinates of targets were obtained by combining color images and depth images from RGB-D cameras. The 3D spatial position of the target in the world coordinate system was affected by the time of the cotton top bud recognition algorithm, the forward speed of the mobile platform and the time consumption of the manipulator moving to the target position. A dynamic compensation method of target coordinates in the moving direction was proposed to ensure the identification and positioning accuracy of cotton top buds in the moving process. To verify the effectiveness of the proposed algorithm, cotton top bud recognition and localization experiments were conducted in the field. The average precision with the proposed improved Cascade R-CNN model was 97.5 %, and the FPS was 13.3 frames per second, which was suitable for different period of a day. The average error of the positioning accuracy in the platform forwarding direction was 4.2 mm, 6.6 mm and 9.6 mm