Sim2real flower detection towards automated Calendula harvesting

Deep learning has gained a lot of attention in the last decade for its use in computer vision. However, a barrier to use deep learning in an agricultural context is the need for large datasets. Agricultural processes are situated in uncontrolled environments, making data collection even harder than in other contexts. Factors such as plant growth, weather conditions, and illumination are largely uncontrolled, making it hard to collect all possible variations in a dataset. This study demonstrates how synthetic generated data can aid to overcome the current barrier and it exemplifies this in the context of automating the detection and localisation of Calendula flowers. To this end, a pipeline was created that utilises photogrammetry and a flower field simulator to create synthetic data of a flower field. Next, the synthetic data is used to train a deep neural network to detect flowers and the transfer from simulation to reality (sim-to-real) is demonstrated on real data. Although the flower detector has not been trained on real data, it reaches an F1 score of up to 86% on the test sets of real data. Subsequently, a stereo vision camera system utilises this detection model to accurately determine the 3D positions of the flowers. The localisation results in an error of 6.9 ± 5.1 mm for the prediction of the flower height. In conclusion, leveraging the potential of synthetic data and sim-to-real capabilities can lower the costs of collecting large datasets in uncontrolled environments and can accelerate the development of precision agricultural applications. [Display omitted] •Limited datasets and large possible variations limit deep learning in agriculture.•Synthetic training data is created using photogrammetry and a game engine.•Detection model is trained on synthetically generated data.•Sim-to-real is validated for the detection of real Calendula flowers.•Localisation of flowers is demonstrated towards automated Calendula harvesting.