Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Pardyl, Adam, Kurzejamski, Grzegorz, Olszewski, Jan, Trzciński, Tomasz, Zieliński, Bartosz
Format: Artikel
Sprache:eng
Schlagworte:
Computer Science - Computer Vision and Pattern Recognition
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Pardyl, Adam
Kurzejamski, Grzegorz
Olszewski, Jan
Trzciński, Tomasz
Zieliński, Bartosz
description Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.
doi_str_mv 10.48550/arxiv.2309.13353
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2309_13353</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2309_13353</sourcerecordid><originalsourceid>FETCH-LOGICAL-a673-8cbcdbdc54a40247fb89b63ae115b455f26cc1e62a1f166f89fc077b0ad4155a3</originalsourceid><addsrcrecordid>eNotz71OwzAUBWAvDKjwAEz4BRLs2NdJ2KBKS6VKHYhYo-s_ZClxIrug9u2hhenonOFIHyEPnJWyAWBPmE7hu6wEa0suBIhbsnl15zlauk3B5mfanZZxTiF-0m7EfAyG7uLydaTvOC3jZfZzoh8hhznSPmHMv31yKd-RG49jdvf_uSL9puvXb8X-sN2tX_YFqloUjdHGamtAomSVrL1uWq0EOs5BSwBfKWO4UxVyz5XyTesNq2vN0EoOgGJFHv9ur5BhSWHCdB4uoOEKEj8aRkZq</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers</title><source>arXiv.org</source><creator>Pardyl, Adam ; Kurzejamski, Grzegorz ; Olszewski, Jan ; Trzciński, Tomasz ; Zieliński, Bartosz</creator><creatorcontrib>Pardyl, Adam ; Kurzejamski, Grzegorz ; Olszewski, Jan ; Trzciński, Tomasz ; Zieliński, Bartosz</creatorcontrib><description>Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.</description><identifier>DOI: 10.48550/arxiv.2309.13353</identifier><language>eng</language><subject>Computer Science - Computer Vision and Pattern Recognition</subject><creationdate>2023-09</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2309.13353$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2309.13353$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Pardyl, Adam</creatorcontrib><creatorcontrib>Kurzejamski, Grzegorz</creatorcontrib><creatorcontrib>Olszewski, Jan</creatorcontrib><creatorcontrib>Trzciński, Tomasz</creatorcontrib><creatorcontrib>Zieliński, Bartosz</creatorcontrib><title>Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers</title><description>Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.</description><subject>Computer Science - Computer Vision and Pattern Recognition</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotz71OwzAUBWAvDKjwAEz4BRLs2NdJ2KBKS6VKHYhYo-s_ZClxIrug9u2hhenonOFIHyEPnJWyAWBPmE7hu6wEa0suBIhbsnl15zlauk3B5mfanZZxTiF-0m7EfAyG7uLydaTvOC3jZfZzoh8hhznSPmHMv31yKd-RG49jdvf_uSL9puvXb8X-sN2tX_YFqloUjdHGamtAomSVrL1uWq0EOs5BSwBfKWO4UxVyz5XyTesNq2vN0EoOgGJFHv9ur5BhSWHCdB4uoOEKEj8aRkZq</recordid><startdate>20230923</startdate><enddate>20230923</enddate><creator>Pardyl, Adam</creator><creator>Kurzejamski, Grzegorz</creator><creator>Olszewski, Jan</creator><creator>Trzciński, Tomasz</creator><creator>Zieliński, Bartosz</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20230923</creationdate><title>Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers</title><author>Pardyl, Adam ; Kurzejamski, Grzegorz ; Olszewski, Jan ; Trzciński, Tomasz ; Zieliński, Bartosz</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a673-8cbcdbdc54a40247fb89b63ae115b455f26cc1e62a1f166f89fc077b0ad4155a3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Computer Science - Computer Vision and Pattern Recognition</topic><toplevel>online_resources</toplevel><creatorcontrib>Pardyl, Adam</creatorcontrib><creatorcontrib>Kurzejamski, Grzegorz</creatorcontrib><creatorcontrib>Olszewski, Jan</creatorcontrib><creatorcontrib>Trzciński, Tomasz</creatorcontrib><creatorcontrib>Zieliński, Bartosz</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Pardyl, Adam</au><au>Kurzejamski, Grzegorz</au><au>Olszewski, Jan</au><au>Trzciński, Tomasz</au><au>Zieliński, Bartosz</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers</atitle><date>2023-09-23</date><risdate>2023</risdate><abstract>Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.</abstract><doi>10.48550/arxiv.2309.13353</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2309.13353
ispartof
issn
language eng
recordid cdi_arxiv_primary_2309_13353
source arXiv.org
subjects Computer Science - Computer Vision and Pattern Recognition
title Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-19T20%3A58%3A44IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Beyond%20Grids:%20Exploring%20Elastic%20Input%20Sampling%20for%20Vision%20Transformers&rft.au=Pardyl,%20Adam&rft.date=2023-09-23&rft_id=info:doi/10.48550/arxiv.2309.13353&rft_dat=%3Carxiv_GOX%3E2309_13353%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true