Optimized Inference Scheme for Conditional Computation in On-Device Object Detection
Recently, conditional computation has been applied to on-device object detection to solve the conflict between huge computation requirments of deep neural network (DNN) and limited computation resources of edge devices. There is a need for an optimized inference scheme that can efficiently perform c...
Gespeichert in:
| Veröffentlicht in: | IEEE embedded systems letters 2024-12, p.1-1 |
|---|---|
| Hauptverfasser: | , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 1 |
|---|---|
| container_issue | |
| container_start_page | 1 |
| container_title | IEEE embedded systems letters |
| container_volume | |
| creator | Zhao, Kairong Chang, Yinghui Wu, Weikang Li, Zirun Luo, Hongyin He, Shan Guo, Donghui |
| description | Recently, conditional computation has been applied to on-device object detection to solve the conflict between huge computation requirments of deep neural network (DNN) and limited computation resources of edge devices. There is a need for an optimized inference scheme that can efficiently perform conditional computation in on-device object detection. This letter proposes a predictor which can predict router decisions of conditional computation. Based on the predictor, this letter also presents an inference scheme which hides router latency through concurrently executing router and the predicted branch. The proposed predictor shows higher accuracy than profiling-based method, and experiment shows that our inference scheme can get latency decrease over traditional scheme. |
| doi_str_mv | 10.1109/LES.2024.3514920 |
| format | Article |
| fullrecord | <record><control><sourceid>ieee_RIE</sourceid><recordid>TN_cdi_ieee_primary_10787219</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>10787219</ieee_id><sourcerecordid>10787219</sourcerecordid><originalsourceid>FETCH-ieee_primary_107872193</originalsourceid><addsrcrecordid>eNqFijsPgjAYRTtoIlF2B4f-AbAPHnYGjSYmDLAThI9YAoWUaqK_3pq4e5dzb85FaEuJTykR--sx9xlhgc9DGghGFsihIuAeiSK-Qu48d8QmDOKQhw4qssnIQb6hwRfVggZVA87rOwyA21HjZFSNNHJUVW_7MD1M9V1YKpwpL4WntP_s1kFtcArGwtoNWrZVP4P74xrtTsciOXsSAMpJy6HSr5KS-BAzKvgf_QHnez_O</addsrcrecordid><sourcetype>Publisher</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Optimized Inference Scheme for Conditional Computation in On-Device Object Detection</title><source>IEEE Electronic Library (IEL)</source><creator>Zhao, Kairong ; Chang, Yinghui ; Wu, Weikang ; Li, Zirun ; Luo, Hongyin ; He, Shan ; Guo, Donghui</creator><creatorcontrib>Zhao, Kairong ; Chang, Yinghui ; Wu, Weikang ; Li, Zirun ; Luo, Hongyin ; He, Shan ; Guo, Donghui</creatorcontrib><description>Recently, conditional computation has been applied to on-device object detection to solve the conflict between huge computation requirments of deep neural network (DNN) and limited computation resources of edge devices. There is a need for an optimized inference scheme that can efficiently perform conditional computation in on-device object detection. This letter proposes a predictor which can predict router decisions of conditional computation. Based on the predictor, this letter also presents an inference scheme which hides router latency through concurrently executing router and the predicted branch. The proposed predictor shows higher accuracy than profiling-based method, and experiment shows that our inference scheme can get latency decrease over traditional scheme.</description><identifier>ISSN: 1943-0663</identifier><identifier>DOI: 10.1109/LES.2024.3514920</identifier><identifier>CODEN: ESLMAP</identifier><language>eng</language><publisher>IEEE</publisher><subject>acceleration ; Accuracy ; Artificial neural networks ; Computational modeling ; conditional computation ; Detectors ; edge AI ; Graphics processing units ; History ; inference optimization ; Object detection ; Predictive models ; Training ; Videos</subject><ispartof>IEEE embedded systems letters, 2024-12, p.1-1</ispartof><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><orcidid>0009-0004-9217-7205 ; 0000-0002-4264-7553 ; 0000-0001-5915-4088</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/10787219$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/10787219$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Zhao, Kairong</creatorcontrib><creatorcontrib>Chang, Yinghui</creatorcontrib><creatorcontrib>Wu, Weikang</creatorcontrib><creatorcontrib>Li, Zirun</creatorcontrib><creatorcontrib>Luo, Hongyin</creatorcontrib><creatorcontrib>He, Shan</creatorcontrib><creatorcontrib>Guo, Donghui</creatorcontrib><title>Optimized Inference Scheme for Conditional Computation in On-Device Object Detection</title><title>IEEE embedded systems letters</title><addtitle>LES</addtitle><description>Recently, conditional computation has been applied to on-device object detection to solve the conflict between huge computation requirments of deep neural network (DNN) and limited computation resources of edge devices. There is a need for an optimized inference scheme that can efficiently perform conditional computation in on-device object detection. This letter proposes a predictor which can predict router decisions of conditional computation. Based on the predictor, this letter also presents an inference scheme which hides router latency through concurrently executing router and the predicted branch. The proposed predictor shows higher accuracy than profiling-based method, and experiment shows that our inference scheme can get latency decrease over traditional scheme.</description><subject>acceleration</subject><subject>Accuracy</subject><subject>Artificial neural networks</subject><subject>Computational modeling</subject><subject>conditional computation</subject><subject>Detectors</subject><subject>edge AI</subject><subject>Graphics processing units</subject><subject>History</subject><subject>inference optimization</subject><subject>Object detection</subject><subject>Predictive models</subject><subject>Training</subject><subject>Videos</subject><issn>1943-0663</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNqFijsPgjAYRTtoIlF2B4f-AbAPHnYGjSYmDLAThI9YAoWUaqK_3pq4e5dzb85FaEuJTykR--sx9xlhgc9DGghGFsihIuAeiSK-Qu48d8QmDOKQhw4qssnIQb6hwRfVggZVA87rOwyA21HjZFSNNHJUVW_7MD1M9V1YKpwpL4WntP_s1kFtcArGwtoNWrZVP4P74xrtTsciOXsSAMpJy6HSr5KS-BAzKvgf_QHnez_O</recordid><startdate>20241209</startdate><enddate>20241209</enddate><creator>Zhao, Kairong</creator><creator>Chang, Yinghui</creator><creator>Wu, Weikang</creator><creator>Li, Zirun</creator><creator>Luo, Hongyin</creator><creator>He, Shan</creator><creator>Guo, Donghui</creator><general>IEEE</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><orcidid>https://orcid.org/0009-0004-9217-7205</orcidid><orcidid>https://orcid.org/0000-0002-4264-7553</orcidid><orcidid>https://orcid.org/0000-0001-5915-4088</orcidid></search><sort><creationdate>20241209</creationdate><title>Optimized Inference Scheme for Conditional Computation in On-Device Object Detection</title><author>Zhao, Kairong ; Chang, Yinghui ; Wu, Weikang ; Li, Zirun ; Luo, Hongyin ; He, Shan ; Guo, Donghui</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-ieee_primary_107872193</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>acceleration</topic><topic>Accuracy</topic><topic>Artificial neural networks</topic><topic>Computational modeling</topic><topic>conditional computation</topic><topic>Detectors</topic><topic>edge AI</topic><topic>Graphics processing units</topic><topic>History</topic><topic>inference optimization</topic><topic>Object detection</topic><topic>Predictive models</topic><topic>Training</topic><topic>Videos</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Zhao, Kairong</creatorcontrib><creatorcontrib>Chang, Yinghui</creatorcontrib><creatorcontrib>Wu, Weikang</creatorcontrib><creatorcontrib>Li, Zirun</creatorcontrib><creatorcontrib>Luo, Hongyin</creatorcontrib><creatorcontrib>He, Shan</creatorcontrib><creatorcontrib>Guo, Donghui</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><jtitle>IEEE embedded systems letters</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Zhao, Kairong</au><au>Chang, Yinghui</au><au>Wu, Weikang</au><au>Li, Zirun</au><au>Luo, Hongyin</au><au>He, Shan</au><au>Guo, Donghui</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Optimized Inference Scheme for Conditional Computation in On-Device Object Detection</atitle><jtitle>IEEE embedded systems letters</jtitle><stitle>LES</stitle><date>2024-12-09</date><risdate>2024</risdate><spage>1</spage><epage>1</epage><pages>1-1</pages><issn>1943-0663</issn><coden>ESLMAP</coden><abstract>Recently, conditional computation has been applied to on-device object detection to solve the conflict between huge computation requirments of deep neural network (DNN) and limited computation resources of edge devices. There is a need for an optimized inference scheme that can efficiently perform conditional computation in on-device object detection. This letter proposes a predictor which can predict router decisions of conditional computation. Based on the predictor, this letter also presents an inference scheme which hides router latency through concurrently executing router and the predicted branch. The proposed predictor shows higher accuracy than profiling-based method, and experiment shows that our inference scheme can get latency decrease over traditional scheme.</abstract><pub>IEEE</pub><doi>10.1109/LES.2024.3514920</doi><orcidid>https://orcid.org/0009-0004-9217-7205</orcidid><orcidid>https://orcid.org/0000-0002-4264-7553</orcidid><orcidid>https://orcid.org/0000-0001-5915-4088</orcidid></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 1943-0663 |
| ispartof | IEEE embedded systems letters, 2024-12, p.1-1 |
| issn | 1943-0663 |
| language | eng |
| recordid | cdi_ieee_primary_10787219 |
| source | IEEE Electronic Library (IEL) |
| subjects | acceleration Accuracy Artificial neural networks Computational modeling conditional computation Detectors edge AI Graphics processing units History inference optimization Object detection Predictive models Training Videos |
| title | Optimized Inference Scheme for Conditional Computation in On-Device Object Detection |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-06T11%3A51%3A31IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-ieee_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Optimized%20Inference%20Scheme%20for%20Conditional%20Computation%20in%20On-Device%20Object%20Detection&rft.jtitle=IEEE%20embedded%20systems%20letters&rft.au=Zhao,%20Kairong&rft.date=2024-12-09&rft.spage=1&rft.epage=1&rft.pages=1-1&rft.issn=1943-0663&rft.coden=ESLMAP&rft_id=info:doi/10.1109/LES.2024.3514920&rft_dat=%3Cieee_RIE%3E10787219%3C/ieee_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rft_ieee_id=10787219&rfr_iscdi=true |