Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss
Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduce...
Gespeichert in:
Veröffentlicht in: | arXiv.org 2022-06 |
---|---|
Hauptverfasser: | , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | López-Cifuentes, Alejandro Escudero-Viñolo, Marcos Bescós, Jesús SanMiguel, Juan C |
description | Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduced via maximum activation maps, as the source knowledge. In this paper, we propose and analyse the use of a 2D frequency transform of the activation maps before transferring them. We pose that\textemdash by using global image cues rather than pixel estimates, this strategy enhances knowledge transferability in tasks such as scene recognition, defined by strong spatial and contextual relationships between multiple and varied concepts. To validate the proposed method, an extensive evaluation of the state-of-the-art in scene recognition is presented. Experimental results provide strong evidences that the proposed strategy enables the student network to better focus on the relevant image areas learnt by the teacher network, hence leading to better descriptive features and higher transferred performance than every other state-of-the-art alternative. We publicly release the training and evaluation framework used along this paper at http://www-vpu.eps.uam.es/publications/DCTBasedKDForSceneRecognition. |
format | Article |
fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2659817099</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2659817099</sourcerecordid><originalsourceid>FETCH-proquest_journals_26598170993</originalsourceid><addsrcrecordid>eNqNyk8PwTAYgPFGIrGw7_Amzk22zv65yUYIbrtPWUen2tnb8fWRcHd6Dr9nQBwWBD5NZoyNiIvYeJ7HopiFYeCQw8Jaoa00mh45igq22jyVqM4CcolWKsU_CFLDvldWUv77oeB4xTkUFwGbW8tPFkwNHPKsoFUnH0LDziBOyLDmCoX77ZhMV8siW9O2M_deoC0b03f6TSWLwjTxYy9Ng_-uF6vmRDI</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2659817099</pqid></control><display><type>article</type><title>Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss</title><source>Free E- Journals</source><creator>López-Cifuentes, Alejandro ; Escudero-Viñolo, Marcos ; Bescós, Jesús ; SanMiguel, Juan C</creator><creatorcontrib>López-Cifuentes, Alejandro ; Escudero-Viñolo, Marcos ; Bescós, Jesús ; SanMiguel, Juan C</creatorcontrib><description>Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduced via maximum activation maps, as the source knowledge. In this paper, we propose and analyse the use of a 2D frequency transform of the activation maps before transferring them. We pose that\textemdash by using global image cues rather than pixel estimates, this strategy enhances knowledge transferability in tasks such as scene recognition, defined by strong spatial and contextual relationships between multiple and varied concepts. To validate the proposed method, an extensive evaluation of the state-of-the-art in scene recognition is presented. Experimental results provide strong evidences that the proposed strategy enables the student network to better focus on the relevant image areas learnt by the teacher network, hence leading to better descriptive features and higher transferred performance than every other state-of-the-art alternative. We publicly release the training and evaluation framework used along this paper at http://www-vpu.eps.uam.es/publications/DCTBasedKDForSceneRecognition.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Artificial neural networks ; Distillation ; Recognition ; Two dimensional analysis</subject><ispartof>arXiv.org, 2022-06</ispartof><rights>2022. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>López-Cifuentes, Alejandro</creatorcontrib><creatorcontrib>Escudero-Viñolo, Marcos</creatorcontrib><creatorcontrib>Bescós, Jesús</creatorcontrib><creatorcontrib>SanMiguel, Juan C</creatorcontrib><title>Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss</title><title>arXiv.org</title><description>Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduced via maximum activation maps, as the source knowledge. In this paper, we propose and analyse the use of a 2D frequency transform of the activation maps before transferring them. We pose that\textemdash by using global image cues rather than pixel estimates, this strategy enhances knowledge transferability in tasks such as scene recognition, defined by strong spatial and contextual relationships between multiple and varied concepts. To validate the proposed method, an extensive evaluation of the state-of-the-art in scene recognition is presented. Experimental results provide strong evidences that the proposed strategy enables the student network to better focus on the relevant image areas learnt by the teacher network, hence leading to better descriptive features and higher transferred performance than every other state-of-the-art alternative. We publicly release the training and evaluation framework used along this paper at http://www-vpu.eps.uam.es/publications/DCTBasedKDForSceneRecognition.</description><subject>Artificial neural networks</subject><subject>Distillation</subject><subject>Recognition</subject><subject>Two dimensional analysis</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNyk8PwTAYgPFGIrGw7_Amzk22zv65yUYIbrtPWUen2tnb8fWRcHd6Dr9nQBwWBD5NZoyNiIvYeJ7HopiFYeCQw8Jaoa00mh45igq22jyVqM4CcolWKsU_CFLDvldWUv77oeB4xTkUFwGbW8tPFkwNHPKsoFUnH0LDziBOyLDmCoX77ZhMV8siW9O2M_deoC0b03f6TSWLwjTxYy9Ng_-uF6vmRDI</recordid><startdate>20220606</startdate><enddate>20220606</enddate><creator>López-Cifuentes, Alejandro</creator><creator>Escudero-Viñolo, Marcos</creator><creator>Bescós, Jesús</creator><creator>SanMiguel, Juan C</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20220606</creationdate><title>Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss</title><author>López-Cifuentes, Alejandro ; Escudero-Viñolo, Marcos ; Bescós, Jesús ; SanMiguel, Juan C</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_26598170993</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Artificial neural networks</topic><topic>Distillation</topic><topic>Recognition</topic><topic>Two dimensional analysis</topic><toplevel>online_resources</toplevel><creatorcontrib>López-Cifuentes, Alejandro</creatorcontrib><creatorcontrib>Escudero-Viñolo, Marcos</creatorcontrib><creatorcontrib>Bescós, Jesús</creatorcontrib><creatorcontrib>SanMiguel, Juan C</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>López-Cifuentes, Alejandro</au><au>Escudero-Viñolo, Marcos</au><au>Bescós, Jesús</au><au>SanMiguel, Juan C</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss</atitle><jtitle>arXiv.org</jtitle><date>2022-06-06</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduced via maximum activation maps, as the source knowledge. In this paper, we propose and analyse the use of a 2D frequency transform of the activation maps before transferring them. We pose that\textemdash by using global image cues rather than pixel estimates, this strategy enhances knowledge transferability in tasks such as scene recognition, defined by strong spatial and contextual relationships between multiple and varied concepts. To validate the proposed method, an extensive evaluation of the state-of-the-art in scene recognition is presented. Experimental results provide strong evidences that the proposed strategy enables the student network to better focus on the relevant image areas learnt by the teacher network, hence leading to better descriptive features and higher transferred performance than every other state-of-the-art alternative. We publicly release the training and evaluation framework used along this paper at http://www-vpu.eps.uam.es/publications/DCTBasedKDForSceneRecognition.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2022-06 |
issn | 2331-8422 |
language | eng |
recordid | cdi_proquest_journals_2659817099 |
source | Free E- Journals |
subjects | Artificial neural networks Distillation Recognition Two dimensional analysis |
title | Attention-based Knowledge Distillation in Multi-attention Tasks: The Impact of a DCT-driven Loss |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T03%3A05%3A08IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Attention-based%20Knowledge%20Distillation%20in%20Multi-attention%20Tasks:%20The%20Impact%20of%20a%20DCT-driven%20Loss&rft.jtitle=arXiv.org&rft.au=L%C3%B3pez-Cifuentes,%20Alejandro&rft.date=2022-06-06&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2659817099%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2659817099&rft_id=info:pmid/&rfr_iscdi=true |