Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation

Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups de...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:arXiv.org 2021-09
Hauptverfasser: Khan, Humair Raj, Gupta, Deepak, Ekbal, Asif
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Khan, Humair Raj
Gupta, Deepak
Ekbal, Asif
description Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.
format Article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2572083209</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2572083209</sourcerecordid><originalsourceid>FETCH-proquest_journals_25720832093</originalsourceid><addsrcrecordid>eNqNjEEKwjAURIMgWNQ7fHBdiInVupSqCOJCLG4lkq-kxkSTxurtbcADuJph5s10SMI4H6f5hLEeGXpfUUrZdMayjCfkVtpGOOlhiS_U9qHMFQTsgq6Vbn0QGoSRUFiJ6U69UcJR-ZjuA_paWQML4xt0cXf4-BrvcP7A1thGo7wiLFVLaS0iOiDdi9Aehz_tk9F6VRab9OHsM96dKhucaasTy2aM5pzROf-P-gJiqUlB</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2572083209</pqid></control><display><type>article</type><title>Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation</title><source>Free E- Journals</source><creator>Khan, Humair Raj ; Gupta, Deepak ; Ekbal, Asif</creator><creatorcontrib>Khan, Humair Raj ; Gupta, Deepak ; Ekbal, Asif</creatorcontrib><description>Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Coders ; Datasets ; Distillation ; English language ; Language ; Multilingualism ; Questions ; Teachers ; Vision ; Visual tasks</subject><ispartof>arXiv.org, 2021-09</ispartof><rights>2021. This work is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>Khan, Humair Raj</creatorcontrib><creatorcontrib>Gupta, Deepak</creatorcontrib><creatorcontrib>Ekbal, Asif</creatorcontrib><title>Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation</title><title>arXiv.org</title><description>Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.</description><subject>Coders</subject><subject>Datasets</subject><subject>Distillation</subject><subject>English language</subject><subject>Language</subject><subject>Multilingualism</subject><subject>Questions</subject><subject>Teachers</subject><subject>Vision</subject><subject>Visual tasks</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNjEEKwjAURIMgWNQ7fHBdiInVupSqCOJCLG4lkq-kxkSTxurtbcADuJph5s10SMI4H6f5hLEeGXpfUUrZdMayjCfkVtpGOOlhiS_U9qHMFQTsgq6Vbn0QGoSRUFiJ6U69UcJR-ZjuA_paWQML4xt0cXf4-BrvcP7A1thGo7wiLFVLaS0iOiDdi9Aehz_tk9F6VRab9OHsM96dKhucaasTy2aM5pzROf-P-gJiqUlB</recordid><startdate>20210910</startdate><enddate>20210910</enddate><creator>Khan, Humair Raj</creator><creator>Gupta, Deepak</creator><creator>Ekbal, Asif</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20210910</creationdate><title>Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation</title><author>Khan, Humair Raj ; Gupta, Deepak ; Ekbal, Asif</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_25720832093</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Coders</topic><topic>Datasets</topic><topic>Distillation</topic><topic>English language</topic><topic>Language</topic><topic>Multilingualism</topic><topic>Questions</topic><topic>Teachers</topic><topic>Vision</topic><topic>Visual tasks</topic><toplevel>online_resources</toplevel><creatorcontrib>Khan, Humair Raj</creatorcontrib><creatorcontrib>Gupta, Deepak</creatorcontrib><creatorcontrib>Ekbal, Asif</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Khan, Humair Raj</au><au>Gupta, Deepak</au><au>Ekbal, Asif</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation</atitle><jtitle>arXiv.org</jtitle><date>2021-09-10</date><risdate>2021</risdate><eissn>2331-8422</eissn><abstract>Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2021-09
issn 2331-8422
language eng
recordid cdi_proquest_journals_2572083209
source Free E- Journals
subjects Coders
Datasets
Distillation
English language
Language
Multilingualism
Questions
Teachers
Vision
Visual tasks
title Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T14%3A09%3A58IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Towards%20Developing%20a%20Multilingual%20and%20Code-Mixed%20Visual%20Question%20Answering%20System%20by%20Knowledge%20Distillation&rft.jtitle=arXiv.org&rft.au=Khan,%20Humair%20Raj&rft.date=2021-09-10&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2572083209%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2572083209&rft_id=info:pmid/&rfr_iscdi=true