SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations

Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements, achieving real-time on-device optical flow estimation remains a complex challenge. First, an optical flow model must be sufficiently lightweight to meet computation and memory constraints to ens...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Lin, Jamie Menjay, Jeong, Jisoo, Cai, Hong, Garrepalli, Risheek, Wang, Kai, Porikli, Fatih
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Lin, Jamie Menjay
Jeong, Jisoo
Cai, Hong
Garrepalli, Risheek
Wang, Kai
Porikli, Fatih
description Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements, achieving real-time on-device optical flow estimation remains a complex challenge. First, an optical flow model must be sufficiently lightweight to meet computation and memory constraints to ensure real-time performance on devices. Second, the necessity for real-time on-device operation imposes constraints that weaken the model's capacity to adequately handle ambiguities in flow estimation, thereby intensifying the difficulty of preserving flow accuracy. This paper introduces two synergistic techniques, Self-Cleaning Iteration (SCI) and Regression Focal Loss (RFL), designed to enhance the capabilities of optical flow models, with a focus on addressing optical flow regression ambiguities. These techniques prove particularly effective in mitigating error propagation, a prevalent issue in optical flow models that employ iterative refinement. Notably, these techniques add negligible to zero overhead in model parameters and inference latency, thereby preserving real-time on-device efficiency. The effectiveness of our proposed SCI and RFL techniques, collectively referred to as SciFlow for brevity, is demonstrated across two distinct lightweight optical flow model architectures in our experiments. Remarkably, SciFlow enables substantial reduction in error metrics (EPE and Fl-all) over the baseline models by up to 6.3% and 10.5% for in-domain scenarios and by up to 6.2% and 13.5% for cross-domain scenarios on the Sintel and KITTI 2015 datasets, respectively.
doi_str_mv 10.48550/arxiv.2404.08135
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2404_08135</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2404_08135</sourcerecordid><originalsourceid>FETCH-LOGICAL-a675-5002d709eeb5439ff9ffdaa94af3890bce6efd2161f1b3d8884946dc020e7ad53</originalsourceid><addsrcrecordid>eNotj0FqwzAURLXpoiQ9QFfVBexKlmRL2RWTtAGXUJK9-ba-EoFiG9nU7e1bp4VhZjMz8Ah55CyVWin2DPHLf6aZZDJlmgt1Tz6Ord-Fft7Q7XXoZ4y-O9PKny_TjIvTwzD5FgJdSvS9txhGOvvpQo8YXFIGhG6Z7CeMMPm-G9fkzkEY8eE_V-S0257Kt6Q6vO7LlyqBvFCJYiyzBTOIjZLCOPcrC2AkOKENa1rM0dmM59zxRlittTQyty3LGBZglViRp7_bG1M9RH-F-F0vbPWNTfwAujBKew</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations</title><source>arXiv.org</source><creator>Lin, Jamie Menjay ; Jeong, Jisoo ; Cai, Hong ; Garrepalli, Risheek ; Wang, Kai ; Porikli, Fatih</creator><creatorcontrib>Lin, Jamie Menjay ; Jeong, Jisoo ; Cai, Hong ; Garrepalli, Risheek ; Wang, Kai ; Porikli, Fatih</creatorcontrib><description>Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements, achieving real-time on-device optical flow estimation remains a complex challenge. First, an optical flow model must be sufficiently lightweight to meet computation and memory constraints to ensure real-time performance on devices. Second, the necessity for real-time on-device operation imposes constraints that weaken the model's capacity to adequately handle ambiguities in flow estimation, thereby intensifying the difficulty of preserving flow accuracy. This paper introduces two synergistic techniques, Self-Cleaning Iteration (SCI) and Regression Focal Loss (RFL), designed to enhance the capabilities of optical flow models, with a focus on addressing optical flow regression ambiguities. These techniques prove particularly effective in mitigating error propagation, a prevalent issue in optical flow models that employ iterative refinement. Notably, these techniques add negligible to zero overhead in model parameters and inference latency, thereby preserving real-time on-device efficiency. The effectiveness of our proposed SCI and RFL techniques, collectively referred to as SciFlow for brevity, is demonstrated across two distinct lightweight optical flow model architectures in our experiments. Remarkably, SciFlow enables substantial reduction in error metrics (EPE and Fl-all) over the baseline models by up to 6.3% and 10.5% for in-domain scenarios and by up to 6.2% and 13.5% for cross-domain scenarios on the Sintel and KITTI 2015 datasets, respectively.</description><identifier>DOI: 10.48550/arxiv.2404.08135</identifier><language>eng</language><subject>Computer Science - Computer Vision and Pattern Recognition</subject><creationdate>2024-04</creationdate><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2404.08135$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2404.08135$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Lin, Jamie Menjay</creatorcontrib><creatorcontrib>Jeong, Jisoo</creatorcontrib><creatorcontrib>Cai, Hong</creatorcontrib><creatorcontrib>Garrepalli, Risheek</creatorcontrib><creatorcontrib>Wang, Kai</creatorcontrib><creatorcontrib>Porikli, Fatih</creatorcontrib><title>SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations</title><description>Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements, achieving real-time on-device optical flow estimation remains a complex challenge. First, an optical flow model must be sufficiently lightweight to meet computation and memory constraints to ensure real-time performance on devices. Second, the necessity for real-time on-device operation imposes constraints that weaken the model's capacity to adequately handle ambiguities in flow estimation, thereby intensifying the difficulty of preserving flow accuracy. This paper introduces two synergistic techniques, Self-Cleaning Iteration (SCI) and Regression Focal Loss (RFL), designed to enhance the capabilities of optical flow models, with a focus on addressing optical flow regression ambiguities. These techniques prove particularly effective in mitigating error propagation, a prevalent issue in optical flow models that employ iterative refinement. Notably, these techniques add negligible to zero overhead in model parameters and inference latency, thereby preserving real-time on-device efficiency. The effectiveness of our proposed SCI and RFL techniques, collectively referred to as SciFlow for brevity, is demonstrated across two distinct lightweight optical flow model architectures in our experiments. Remarkably, SciFlow enables substantial reduction in error metrics (EPE and Fl-all) over the baseline models by up to 6.3% and 10.5% for in-domain scenarios and by up to 6.2% and 13.5% for cross-domain scenarios on the Sintel and KITTI 2015 datasets, respectively.</description><subject>Computer Science - Computer Vision and Pattern Recognition</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj0FqwzAURLXpoiQ9QFfVBexKlmRL2RWTtAGXUJK9-ba-EoFiG9nU7e1bp4VhZjMz8Ah55CyVWin2DPHLf6aZZDJlmgt1Tz6Ord-Fft7Q7XXoZ4y-O9PKny_TjIvTwzD5FgJdSvS9txhGOvvpQo8YXFIGhG6Z7CeMMPm-G9fkzkEY8eE_V-S0257Kt6Q6vO7LlyqBvFCJYiyzBTOIjZLCOPcrC2AkOKENa1rM0dmM59zxRlittTQyty3LGBZglViRp7_bG1M9RH-F-F0vbPWNTfwAujBKew</recordid><startdate>20240411</startdate><enddate>20240411</enddate><creator>Lin, Jamie Menjay</creator><creator>Jeong, Jisoo</creator><creator>Cai, Hong</creator><creator>Garrepalli, Risheek</creator><creator>Wang, Kai</creator><creator>Porikli, Fatih</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20240411</creationdate><title>SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations</title><author>Lin, Jamie Menjay ; Jeong, Jisoo ; Cai, Hong ; Garrepalli, Risheek ; Wang, Kai ; Porikli, Fatih</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a675-5002d709eeb5439ff9ffdaa94af3890bce6efd2161f1b3d8884946dc020e7ad53</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Computer Vision and Pattern Recognition</topic><toplevel>online_resources</toplevel><creatorcontrib>Lin, Jamie Menjay</creatorcontrib><creatorcontrib>Jeong, Jisoo</creatorcontrib><creatorcontrib>Cai, Hong</creatorcontrib><creatorcontrib>Garrepalli, Risheek</creatorcontrib><creatorcontrib>Wang, Kai</creatorcontrib><creatorcontrib>Porikli, Fatih</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Lin, Jamie Menjay</au><au>Jeong, Jisoo</au><au>Cai, Hong</au><au>Garrepalli, Risheek</au><au>Wang, Kai</au><au>Porikli, Fatih</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations</atitle><date>2024-04-11</date><risdate>2024</risdate><abstract>Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements, achieving real-time on-device optical flow estimation remains a complex challenge. First, an optical flow model must be sufficiently lightweight to meet computation and memory constraints to ensure real-time performance on devices. Second, the necessity for real-time on-device operation imposes constraints that weaken the model's capacity to adequately handle ambiguities in flow estimation, thereby intensifying the difficulty of preserving flow accuracy. This paper introduces two synergistic techniques, Self-Cleaning Iteration (SCI) and Regression Focal Loss (RFL), designed to enhance the capabilities of optical flow models, with a focus on addressing optical flow regression ambiguities. These techniques prove particularly effective in mitigating error propagation, a prevalent issue in optical flow models that employ iterative refinement. Notably, these techniques add negligible to zero overhead in model parameters and inference latency, thereby preserving real-time on-device efficiency. The effectiveness of our proposed SCI and RFL techniques, collectively referred to as SciFlow for brevity, is demonstrated across two distinct lightweight optical flow model architectures in our experiments. Remarkably, SciFlow enables substantial reduction in error metrics (EPE and Fl-all) over the baseline models by up to 6.3% and 10.5% for in-domain scenarios and by up to 6.2% and 13.5% for cross-domain scenarios on the Sintel and KITTI 2015 datasets, respectively.</abstract><doi>10.48550/arxiv.2404.08135</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2404.08135
ispartof
issn
language eng
recordid cdi_arxiv_primary_2404_08135
source arXiv.org
subjects Computer Science - Computer Vision and Pattern Recognition
title SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T07%3A15%3A26IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=SciFlow:%20Empowering%20Lightweight%20Optical%20Flow%20Models%20with%20Self-Cleaning%20Iterations&rft.au=Lin,%20Jamie%20Menjay&rft.date=2024-04-11&rft_id=info:doi/10.48550/arxiv.2404.08135&rft_dat=%3Carxiv_GOX%3E2404_08135%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true