How transfer learning impacts linguistic knowledge in deep NLP models?
Transfer learning from pre-trained neural language models towards downstream tasks has been a predominant theme in NLP recently. Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. We investigate how fine-tu...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | Durrani, Nadir Sajjad, Hassan Dalvi, Fahim |
description | Transfer learning from pre-trained neural language models towards downstream
tasks has been a predominant theme in NLP recently. Several researchers have
shown that deep NLP models learn non-trivial amount of linguistic knowledge,
captured at different layers of the model. We investigate how fine-tuning
towards downstream NLP tasks impacts the learned linguistic knowledge. We carry
out a study across popular pre-trained models BERT, RoBERTa and XLNet using
layer and neuron-level diagnostic classifiers. We found that for some GLUE
tasks, the network relies on the core linguistic information and preserve it
deeper in the network, while for others it forgets. Linguistic information is
distributed in the pre-trained language models but becomes localized to the
lower layers post fine-tuning, reserving higher layers for the task specific
knowledge. The pattern varies across architectures, with BERT retaining
linguistic information relatively deeper in the network compared to RoBERTa and
XLNet, where it is predominantly delegated to the lower layers. |
doi_str_mv | 10.48550/arxiv.2105.15179 |
format | Article |
fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2105_15179</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2105_15179</sourcerecordid><originalsourceid>FETCH-LOGICAL-a679-c553af6f1b1f844c1dd4c256299c9ef56ab9c55fdaa86e7e50ec6ac1abd890f83</originalsourceid><addsrcrecordid>eNotz7FOwzAYBGAvDKjwAEz4BRLsJHbsqUIVpUgRMHSP_ti_K6uOE9mBwttTCtPppNNJHyF3nJWNEoI9QPryn2XFmSi54K2-JtvddKJLgpgdJhoQUvTxQP04g1kyDefy4fPiDT3G6RTQHpD6SC3iTF-7dzpOFkNe35ArByHj7X-uyH77tN_siu7t-WXz2BUgW10YIWpw0vGBO9U0hlvbmErISmuj0QkJgz5vnAVQElsUDI0Ew2GwSjOn6hW5_7u9QPo5-RHSd_8L6i-g-gdQcEcU</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>How transfer learning impacts linguistic knowledge in deep NLP models?</title><source>arXiv.org</source><creator>Durrani, Nadir ; Sajjad, Hassan ; Dalvi, Fahim</creator><creatorcontrib>Durrani, Nadir ; Sajjad, Hassan ; Dalvi, Fahim</creatorcontrib><description>Transfer learning from pre-trained neural language models towards downstream
tasks has been a predominant theme in NLP recently. Several researchers have
shown that deep NLP models learn non-trivial amount of linguistic knowledge,
captured at different layers of the model. We investigate how fine-tuning
towards downstream NLP tasks impacts the learned linguistic knowledge. We carry
out a study across popular pre-trained models BERT, RoBERTa and XLNet using
layer and neuron-level diagnostic classifiers. We found that for some GLUE
tasks, the network relies on the core linguistic information and preserve it
deeper in the network, while for others it forgets. Linguistic information is
distributed in the pre-trained language models but becomes localized to the
lower layers post fine-tuning, reserving higher layers for the task specific
knowledge. The pattern varies across architectures, with BERT retaining
linguistic information relatively deeper in the network compared to RoBERTa and
XLNet, where it is predominantly delegated to the lower layers.</description><identifier>DOI: 10.48550/arxiv.2105.15179</identifier><language>eng</language><subject>Computer Science - Computation and Language</subject><creationdate>2021-05</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2105.15179$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2105.15179$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Durrani, Nadir</creatorcontrib><creatorcontrib>Sajjad, Hassan</creatorcontrib><creatorcontrib>Dalvi, Fahim</creatorcontrib><title>How transfer learning impacts linguistic knowledge in deep NLP models?</title><description>Transfer learning from pre-trained neural language models towards downstream
tasks has been a predominant theme in NLP recently. Several researchers have
shown that deep NLP models learn non-trivial amount of linguistic knowledge,
captured at different layers of the model. We investigate how fine-tuning
towards downstream NLP tasks impacts the learned linguistic knowledge. We carry
out a study across popular pre-trained models BERT, RoBERTa and XLNet using
layer and neuron-level diagnostic classifiers. We found that for some GLUE
tasks, the network relies on the core linguistic information and preserve it
deeper in the network, while for others it forgets. Linguistic information is
distributed in the pre-trained language models but becomes localized to the
lower layers post fine-tuning, reserving higher layers for the task specific
knowledge. The pattern varies across architectures, with BERT retaining
linguistic information relatively deeper in the network compared to RoBERTa and
XLNet, where it is predominantly delegated to the lower layers.</description><subject>Computer Science - Computation and Language</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotz7FOwzAYBGAvDKjwAEz4BRLsJHbsqUIVpUgRMHSP_ti_K6uOE9mBwttTCtPppNNJHyF3nJWNEoI9QPryn2XFmSi54K2-JtvddKJLgpgdJhoQUvTxQP04g1kyDefy4fPiDT3G6RTQHpD6SC3iTF-7dzpOFkNe35ArByHj7X-uyH77tN_siu7t-WXz2BUgW10YIWpw0vGBO9U0hlvbmErISmuj0QkJgz5vnAVQElsUDI0Ew2GwSjOn6hW5_7u9QPo5-RHSd_8L6i-g-gdQcEcU</recordid><startdate>20210531</startdate><enddate>20210531</enddate><creator>Durrani, Nadir</creator><creator>Sajjad, Hassan</creator><creator>Dalvi, Fahim</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20210531</creationdate><title>How transfer learning impacts linguistic knowledge in deep NLP models?</title><author>Durrani, Nadir ; Sajjad, Hassan ; Dalvi, Fahim</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a679-c553af6f1b1f844c1dd4c256299c9ef56ab9c55fdaa86e7e50ec6ac1abd890f83</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Computer Science - Computation and Language</topic><toplevel>online_resources</toplevel><creatorcontrib>Durrani, Nadir</creatorcontrib><creatorcontrib>Sajjad, Hassan</creatorcontrib><creatorcontrib>Dalvi, Fahim</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Durrani, Nadir</au><au>Sajjad, Hassan</au><au>Dalvi, Fahim</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>How transfer learning impacts linguistic knowledge in deep NLP models?</atitle><date>2021-05-31</date><risdate>2021</risdate><abstract>Transfer learning from pre-trained neural language models towards downstream
tasks has been a predominant theme in NLP recently. Several researchers have
shown that deep NLP models learn non-trivial amount of linguistic knowledge,
captured at different layers of the model. We investigate how fine-tuning
towards downstream NLP tasks impacts the learned linguistic knowledge. We carry
out a study across popular pre-trained models BERT, RoBERTa and XLNet using
layer and neuron-level diagnostic classifiers. We found that for some GLUE
tasks, the network relies on the core linguistic information and preserve it
deeper in the network, while for others it forgets. Linguistic information is
distributed in the pre-trained language models but becomes localized to the
lower layers post fine-tuning, reserving higher layers for the task specific
knowledge. The pattern varies across architectures, with BERT retaining
linguistic information relatively deeper in the network compared to RoBERTa and
XLNet, where it is predominantly delegated to the lower layers.</abstract><doi>10.48550/arxiv.2105.15179</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | DOI: 10.48550/arxiv.2105.15179 |
ispartof | |
issn | |
language | eng |
recordid | cdi_arxiv_primary_2105_15179 |
source | arXiv.org |
subjects | Computer Science - Computation and Language |
title | How transfer learning impacts linguistic knowledge in deep NLP models? |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-29T18%3A25%3A40IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=How%20transfer%20learning%20impacts%20linguistic%20knowledge%20in%20deep%20NLP%20models?&rft.au=Durrani,%20Nadir&rft.date=2021-05-31&rft_id=info:doi/10.48550/arxiv.2105.15179&rft_dat=%3Carxiv_GOX%3E2105_15179%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |