Stem localization of sweet-pepper plants using the support wire as a visual cue

•First study of stem localization under varying lighting conditions.•Use of support wires as a visual cue to localize the stem.•Novel algorithm employs adaptive thresholding, and object-based and 3D features.•True- and false-positive detection rate of stems 94% and 5%.•Accuracy of stereo-vision under greenhouse conditions needs improvement. A robot arm should avoid collisions with the plant stem when it approaches a candidate sweet-pepper for harvesting. This study therefore aims at stem localization, a topic so far only studied under controlled lighting conditions. Objectives were to develop an algorithm capable of stem localization, using detection of the support wire that is twisted around the stem; to quantitatively evaluate performance of wire detection and stem localization under varying lighting conditions; to determine depth accuracy of stereo-vision under lab and greenhouse conditions. A single colour camera was mounted on a pneumatic slide to record image pairs with a small baseline of 1cm. Artificial lighting was developed to mitigate disturbances caused by natural lighting conditions. An algorithm consisting of five steps was developed and includes novel components such as adaptive thresholding, use of support wires as a visual cue, use of object-based and 3D features and use of minimum expected stem distance. Wire detection rates (true-positive/scaled false-positive) were more favourable under moderate irradiance (94/5%) than under strong irradiance (74/26%). Error of stem localization was measured, in the horizontal plane, by Euclidean distance. Error was smaller for interpolated segments (0.8cm), where a support wire was detected, than for extrapolated segments (1.5cm), where a support wire was not detected. Error increased under strong irradiance. Accuracy of the stereo-vision system (±0.4cm) met the requirements (±1cm) in the lab, but not in the greenhouse (±4.5cm) due to plant movement during recording. The algorithm is probably capable to construct a useful collision map for robotic harvesting, if the issue of inaccurate stereo-vision can be resolved by directions proposed for future work. This is the first study regarding stem localization under varying lighting conditions, and can be useful for future applications in crops that grow along a support wire.