Sim2real flower detection towards automated Calendula harvesting

Deep learning has gained a lot of attention in the last decade for its use in computer vision. However, a barrier to use deep learning in an agricultural context is the need for large datasets. Agricultural processes are situated in uncontrolled environments, making data collection even harder than...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Biosystems engineering 2023-10, Vol.234, p.125-139
Hauptverfasser: Vierbergen, Wout, Willekens, Axel, Dekeyser, Donald, Cool, Simon, wyffels, Francis
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 139
container_issue
container_start_page 125
container_title Biosystems engineering
container_volume 234
creator Vierbergen, Wout
Willekens, Axel
Dekeyser, Donald
Cool, Simon
wyffels, Francis
description Deep learning has gained a lot of attention in the last decade for its use in computer vision. However, a barrier to use deep learning in an agricultural context is the need for large datasets. Agricultural processes are situated in uncontrolled environments, making data collection even harder than in other contexts. Factors such as plant growth, weather conditions, and illumination are largely uncontrolled, making it hard to collect all possible variations in a dataset. This study demonstrates how synthetic generated data can aid to overcome the current barrier and it exemplifies this in the context of automating the detection and localisation of Calendula flowers. To this end, a pipeline was created that utilises photogrammetry and a flower field simulator to create synthetic data of a flower field. Next, the synthetic data is used to train a deep neural network to detect flowers and the transfer from simulation to reality (sim-to-real) is demonstrated on real data. Although the flower detector has not been trained on real data, it reaches an F1 score of up to 86% on the test sets of real data. Subsequently, a stereo vision camera system utilises this detection model to accurately determine the 3D positions of the flowers. The localisation results in an error of 6.9 ± 5.1 mm for the prediction of the flower height. In conclusion, leveraging the potential of synthetic data and sim-to-real capabilities can lower the costs of collecting large datasets in uncontrolled environments and can accelerate the development of precision agricultural applications. [Display omitted] •Limited datasets and large possible variations limit deep learning in agriculture.•Synthetic training data is created using photogrammetry and a game engine.•Detection model is trained on synthetically generated data.•Sim-to-real is validated for the detection of real Calendula flowers.•Localisation of flowers is demonstrated towards automated Calendula harvesting.
doi_str_mv 10.1016/j.biosystemseng.2023.08.016
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_miscellaneous_3153177681</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><els_id>S1537511023001770</els_id><sourcerecordid>3153177681</sourcerecordid><originalsourceid>FETCH-LOGICAL-c355t-11fb5bb6e95338eb22c07737b9c2a75fad11d5d73fa399e18491859de67518bb3</originalsourceid><addsrcrecordid>eNqNkEtLw0AQx4MoWKvfIeDFS-JOtptN8KKU-oCCB_W87GNStyTZurtp6bc3pSJ48zQD_8cwvyS5BpIDgfJ2nSvrwj5E7AL2q7wgBc1JlY_aSTIBRnnGoKhPf3cg58lFCGtCgPFZOUnu32xXeJRt2rRuhz41GFFH6_o0up30JqRyiK6TEU06ly32Zmhl-in9FkO0_eoyOWtkG_DqZ06Tj8fF-_w5W74-vcwflpmmjMUMoFFMqRJrRmmFqig04ZxyVetCctZIA2CY4bSRtK4RqlkNFasNlpxBpRSdJjfH3o13X8N4W3Q2aGxb2aMbgqDjh8B5WcFovTtatXcheGzExttO-r0AIg7cxFr84SYO3ASpxKiN6cUxjeM3W4teBG2x12isH8kI4-y_er4BaON_YA</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3153177681</pqid></control><display><type>article</type><title>Sim2real flower detection towards automated Calendula harvesting</title><source>Elsevier ScienceDirect Journals</source><creator>Vierbergen, Wout ; Willekens, Axel ; Dekeyser, Donald ; Cool, Simon ; wyffels, Francis</creator><creatorcontrib>Vierbergen, Wout ; Willekens, Axel ; Dekeyser, Donald ; Cool, Simon ; wyffels, Francis</creatorcontrib><description>Deep learning has gained a lot of attention in the last decade for its use in computer vision. However, a barrier to use deep learning in an agricultural context is the need for large datasets. Agricultural processes are situated in uncontrolled environments, making data collection even harder than in other contexts. Factors such as plant growth, weather conditions, and illumination are largely uncontrolled, making it hard to collect all possible variations in a dataset. This study demonstrates how synthetic generated data can aid to overcome the current barrier and it exemplifies this in the context of automating the detection and localisation of Calendula flowers. To this end, a pipeline was created that utilises photogrammetry and a flower field simulator to create synthetic data of a flower field. Next, the synthetic data is used to train a deep neural network to detect flowers and the transfer from simulation to reality (sim-to-real) is demonstrated on real data. Although the flower detector has not been trained on real data, it reaches an F1 score of up to 86% on the test sets of real data. Subsequently, a stereo vision camera system utilises this detection model to accurately determine the 3D positions of the flowers. The localisation results in an error of 6.9 ± 5.1 mm for the prediction of the flower height. In conclusion, leveraging the potential of synthetic data and sim-to-real capabilities can lower the costs of collecting large datasets in uncontrolled environments and can accelerate the development of precision agricultural applications. [Display omitted] •Limited datasets and large possible variations limit deep learning in agriculture.•Synthetic training data is created using photogrammetry and a game engine.•Detection model is trained on synthetically generated data.•Sim-to-real is validated for the detection of real Calendula flowers.•Localisation of flowers is demonstrated towards automated Calendula harvesting.</description><identifier>ISSN: 1537-5110</identifier><identifier>EISSN: 1537-5129</identifier><identifier>DOI: 10.1016/j.biosystemseng.2023.08.016</identifier><language>eng</language><publisher>Elsevier Ltd</publisher><subject>automation ; Calendula ; cameras ; computer vision ; data collection ; Deep learning ; flowers ; lighting ; photogrammetry ; plant growth ; Precision agriculture ; prediction ; Sim-to-real ; Synthetic data ; weather</subject><ispartof>Biosystems engineering, 2023-10, Vol.234, p.125-139</ispartof><rights>2023 IAgrE</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c355t-11fb5bb6e95338eb22c07737b9c2a75fad11d5d73fa399e18491859de67518bb3</cites><orcidid>0000-0001-5933-6869 ; 0000-0002-5187-0423 ; 0000-0002-5491-8349 ; 0000-0002-3893-5632</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.sciencedirect.com/science/article/pii/S1537511023001770$$EHTML$$P50$$Gelsevier$$H</linktohtml><link.rule.ids>314,776,780,3537,27901,27902,65306</link.rule.ids></links><search><creatorcontrib>Vierbergen, Wout</creatorcontrib><creatorcontrib>Willekens, Axel</creatorcontrib><creatorcontrib>Dekeyser, Donald</creatorcontrib><creatorcontrib>Cool, Simon</creatorcontrib><creatorcontrib>wyffels, Francis</creatorcontrib><title>Sim2real flower detection towards automated Calendula harvesting</title><title>Biosystems engineering</title><description>Deep learning has gained a lot of attention in the last decade for its use in computer vision. However, a barrier to use deep learning in an agricultural context is the need for large datasets. Agricultural processes are situated in uncontrolled environments, making data collection even harder than in other contexts. Factors such as plant growth, weather conditions, and illumination are largely uncontrolled, making it hard to collect all possible variations in a dataset. This study demonstrates how synthetic generated data can aid to overcome the current barrier and it exemplifies this in the context of automating the detection and localisation of Calendula flowers. To this end, a pipeline was created that utilises photogrammetry and a flower field simulator to create synthetic data of a flower field. Next, the synthetic data is used to train a deep neural network to detect flowers and the transfer from simulation to reality (sim-to-real) is demonstrated on real data. Although the flower detector has not been trained on real data, it reaches an F1 score of up to 86% on the test sets of real data. Subsequently, a stereo vision camera system utilises this detection model to accurately determine the 3D positions of the flowers. The localisation results in an error of 6.9 ± 5.1 mm for the prediction of the flower height. In conclusion, leveraging the potential of synthetic data and sim-to-real capabilities can lower the costs of collecting large datasets in uncontrolled environments and can accelerate the development of precision agricultural applications. [Display omitted] •Limited datasets and large possible variations limit deep learning in agriculture.•Synthetic training data is created using photogrammetry and a game engine.•Detection model is trained on synthetically generated data.•Sim-to-real is validated for the detection of real Calendula flowers.•Localisation of flowers is demonstrated towards automated Calendula harvesting.</description><subject>automation</subject><subject>Calendula</subject><subject>cameras</subject><subject>computer vision</subject><subject>data collection</subject><subject>Deep learning</subject><subject>flowers</subject><subject>lighting</subject><subject>photogrammetry</subject><subject>plant growth</subject><subject>Precision agriculture</subject><subject>prediction</subject><subject>Sim-to-real</subject><subject>Synthetic data</subject><subject>weather</subject><issn>1537-5110</issn><issn>1537-5129</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><recordid>eNqNkEtLw0AQx4MoWKvfIeDFS-JOtptN8KKU-oCCB_W87GNStyTZurtp6bc3pSJ48zQD_8cwvyS5BpIDgfJ2nSvrwj5E7AL2q7wgBc1JlY_aSTIBRnnGoKhPf3cg58lFCGtCgPFZOUnu32xXeJRt2rRuhz41GFFH6_o0up30JqRyiK6TEU06ly32Zmhl-in9FkO0_eoyOWtkG_DqZ06Tj8fF-_w5W74-vcwflpmmjMUMoFFMqRJrRmmFqig04ZxyVetCctZIA2CY4bSRtK4RqlkNFasNlpxBpRSdJjfH3o13X8N4W3Q2aGxb2aMbgqDjh8B5WcFovTtatXcheGzExttO-r0AIg7cxFr84SYO3ASpxKiN6cUxjeM3W4teBG2x12isH8kI4-y_er4BaON_YA</recordid><startdate>202310</startdate><enddate>202310</enddate><creator>Vierbergen, Wout</creator><creator>Willekens, Axel</creator><creator>Dekeyser, Donald</creator><creator>Cool, Simon</creator><creator>wyffels, Francis</creator><general>Elsevier Ltd</general><scope>AAYXX</scope><scope>CITATION</scope><scope>7S9</scope><scope>L.6</scope><orcidid>https://orcid.org/0000-0001-5933-6869</orcidid><orcidid>https://orcid.org/0000-0002-5187-0423</orcidid><orcidid>https://orcid.org/0000-0002-5491-8349</orcidid><orcidid>https://orcid.org/0000-0002-3893-5632</orcidid></search><sort><creationdate>202310</creationdate><title>Sim2real flower detection towards automated Calendula harvesting</title><author>Vierbergen, Wout ; Willekens, Axel ; Dekeyser, Donald ; Cool, Simon ; wyffels, Francis</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c355t-11fb5bb6e95338eb22c07737b9c2a75fad11d5d73fa399e18491859de67518bb3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>automation</topic><topic>Calendula</topic><topic>cameras</topic><topic>computer vision</topic><topic>data collection</topic><topic>Deep learning</topic><topic>flowers</topic><topic>lighting</topic><topic>photogrammetry</topic><topic>plant growth</topic><topic>Precision agriculture</topic><topic>prediction</topic><topic>Sim-to-real</topic><topic>Synthetic data</topic><topic>weather</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Vierbergen, Wout</creatorcontrib><creatorcontrib>Willekens, Axel</creatorcontrib><creatorcontrib>Dekeyser, Donald</creatorcontrib><creatorcontrib>Cool, Simon</creatorcontrib><creatorcontrib>wyffels, Francis</creatorcontrib><collection>CrossRef</collection><collection>AGRICOLA</collection><collection>AGRICOLA - Academic</collection><jtitle>Biosystems engineering</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Vierbergen, Wout</au><au>Willekens, Axel</au><au>Dekeyser, Donald</au><au>Cool, Simon</au><au>wyffels, Francis</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Sim2real flower detection towards automated Calendula harvesting</atitle><jtitle>Biosystems engineering</jtitle><date>2023-10</date><risdate>2023</risdate><volume>234</volume><spage>125</spage><epage>139</epage><pages>125-139</pages><issn>1537-5110</issn><eissn>1537-5129</eissn><abstract>Deep learning has gained a lot of attention in the last decade for its use in computer vision. However, a barrier to use deep learning in an agricultural context is the need for large datasets. Agricultural processes are situated in uncontrolled environments, making data collection even harder than in other contexts. Factors such as plant growth, weather conditions, and illumination are largely uncontrolled, making it hard to collect all possible variations in a dataset. This study demonstrates how synthetic generated data can aid to overcome the current barrier and it exemplifies this in the context of automating the detection and localisation of Calendula flowers. To this end, a pipeline was created that utilises photogrammetry and a flower field simulator to create synthetic data of a flower field. Next, the synthetic data is used to train a deep neural network to detect flowers and the transfer from simulation to reality (sim-to-real) is demonstrated on real data. Although the flower detector has not been trained on real data, it reaches an F1 score of up to 86% on the test sets of real data. Subsequently, a stereo vision camera system utilises this detection model to accurately determine the 3D positions of the flowers. The localisation results in an error of 6.9 ± 5.1 mm for the prediction of the flower height. In conclusion, leveraging the potential of synthetic data and sim-to-real capabilities can lower the costs of collecting large datasets in uncontrolled environments and can accelerate the development of precision agricultural applications. [Display omitted] •Limited datasets and large possible variations limit deep learning in agriculture.•Synthetic training data is created using photogrammetry and a game engine.•Detection model is trained on synthetically generated data.•Sim-to-real is validated for the detection of real Calendula flowers.•Localisation of flowers is demonstrated towards automated Calendula harvesting.</abstract><pub>Elsevier Ltd</pub><doi>10.1016/j.biosystemseng.2023.08.016</doi><tpages>15</tpages><orcidid>https://orcid.org/0000-0001-5933-6869</orcidid><orcidid>https://orcid.org/0000-0002-5187-0423</orcidid><orcidid>https://orcid.org/0000-0002-5491-8349</orcidid><orcidid>https://orcid.org/0000-0002-3893-5632</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 1537-5110
ispartof Biosystems engineering, 2023-10, Vol.234, p.125-139
issn 1537-5110
1537-5129
language eng
recordid cdi_proquest_miscellaneous_3153177681
source Elsevier ScienceDirect Journals
subjects automation
Calendula
cameras
computer vision
data collection
Deep learning
flowers
lighting
photogrammetry
plant growth
Precision agriculture
prediction
Sim-to-real
Synthetic data
weather
title Sim2real flower detection towards automated Calendula harvesting
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-31T22%3A30%3A08IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Sim2real%20flower%20detection%20towards%20automated%20Calendula%20harvesting&rft.jtitle=Biosystems%20engineering&rft.au=Vierbergen,%20Wout&rft.date=2023-10&rft.volume=234&rft.spage=125&rft.epage=139&rft.pages=125-139&rft.issn=1537-5110&rft.eissn=1537-5129&rft_id=info:doi/10.1016/j.biosystemseng.2023.08.016&rft_dat=%3Cproquest_cross%3E3153177681%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3153177681&rft_id=info:pmid/&rft_els_id=S1537511023001770&rfr_iscdi=true