Pre-trained models for natural language processing: A survey

Pre-trained models for natural language processing: A survey

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Science China. Technological sciences 2020-10, Vol.63 (10), p.1872-1897
Hauptverfasser: Qiu, XiPeng, Sun, TianXiang, Xu, YiGe, Shao, YunFan, Dai, Ning, Huang, XuanJing
Format: Artikel
Sprache:eng
Schlagworte:
Engineering
Natural language processing
Review
Taxonomy
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 1897
container_issue 10
container_start_page 1872
container_title Science China. Technological sciences
container_volume 63
creator Qiu, XiPeng
Sun, TianXiang
Xu, YiGe
Shao, YunFan
Dai, Ning
Huang, XuanJing
description Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
doi_str_mv 10.1007/s11431-020-1647-3
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2452108732</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2452108732</sourcerecordid><originalsourceid>FETCH-LOGICAL-c359t-91787d931a77f9402479bd2e85bcf7da44d462defd558c3240547253e8969203</originalsourceid><addsrcrecordid>eNp1kE1LAzEQhoMoWGp_gLeA5-hMPjYb8VKKX1DQQ-8h3WSXlu1uTXaF_ntTVvDkXGYO7_vOzEPILcI9AuiHhCgFMuDAsJCaiQsyw7IwDA3AZZ4LLZkWHK_JIqU95BKlAZQz8vQZAxui23XB00PvQ5to3UfauWGMrqWt65rRNYEeY1-FlHZd80iXNI3xO5xuyFXt2hQWv31ONi_Pm9UbW3-8vq-Wa1YJZQZmUJfaG4FO69pI4FKbreehVNuq1t5J6WXBfai9UmUluAQlNVcilKYwHMSc3E2x-YavMaTB7vsxdnmj5VJxhDL_llU4qarYpxRDbY9xd3DxZBHsGZOdMNmMyZ4xWZE9fPKkrO2aEP-S_zf9AGgvaBA</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2452108732</pqid></control><display><type>article</type><title>Pre-trained models for natural language processing: A survey</title><source>Alma/SFX Local Collection</source><source>SpringerLink Journals - AutoHoldings</source><creator>Qiu, XiPeng ; Sun, TianXiang ; Xu, YiGe ; Shao, YunFan ; Dai, Ning ; Huang, XuanJing</creator><creatorcontrib>Qiu, XiPeng ; Sun, TianXiang ; Xu, YiGe ; Shao, YunFan ; Dai, Ning ; Huang, XuanJing</creatorcontrib><description>Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.</description><identifier>ISSN: 1674-7321</identifier><identifier>EISSN: 1869-1900</identifier><identifier>DOI: 10.1007/s11431-020-1647-3</identifier><language>eng</language><publisher>Beijing: Science China Press</publisher><subject>Engineering ; Natural language processing ; Review ; Taxonomy</subject><ispartof>Science China. Technological sciences, 2020-10, Vol.63 (10), p.1872-1897</ispartof><rights>Science China Press and Springer-Verlag GmbH Germany, part of Springer Nature 2020</rights><rights>Science China Press and Springer-Verlag GmbH Germany, part of Springer Nature 2020.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c359t-91787d931a77f9402479bd2e85bcf7da44d462defd558c3240547253e8969203</citedby><cites>FETCH-LOGICAL-c359t-91787d931a77f9402479bd2e85bcf7da44d462defd558c3240547253e8969203</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://link.springer.com/content/pdf/10.1007/s11431-020-1647-3$$EPDF$$P50$$Gspringer$$H</linktopdf><linktohtml>$$Uhttps://link.springer.com/10.1007/s11431-020-1647-3$$EHTML$$P50$$Gspringer$$H</linktohtml><link.rule.ids>314,776,780,27901,27902,41464,42533,51294</link.rule.ids></links><search><creatorcontrib>Qiu, XiPeng</creatorcontrib><creatorcontrib>Sun, TianXiang</creatorcontrib><creatorcontrib>Xu, YiGe</creatorcontrib><creatorcontrib>Shao, YunFan</creatorcontrib><creatorcontrib>Dai, Ning</creatorcontrib><creatorcontrib>Huang, XuanJing</creatorcontrib><title>Pre-trained models for natural language processing: A survey</title><title>Science China. Technological sciences</title><addtitle>Sci. China Technol. Sci</addtitle><description>Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.</description><subject>Engineering</subject><subject>Natural language processing</subject><subject>Review</subject><subject>Taxonomy</subject><issn>1674-7321</issn><issn>1869-1900</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2020</creationdate><recordtype>article</recordtype><recordid>eNp1kE1LAzEQhoMoWGp_gLeA5-hMPjYb8VKKX1DQQ-8h3WSXlu1uTXaF_ntTVvDkXGYO7_vOzEPILcI9AuiHhCgFMuDAsJCaiQsyw7IwDA3AZZ4LLZkWHK_JIqU95BKlAZQz8vQZAxui23XB00PvQ5to3UfauWGMrqWt65rRNYEeY1-FlHZd80iXNI3xO5xuyFXt2hQWv31ONi_Pm9UbW3-8vq-Wa1YJZQZmUJfaG4FO69pI4FKbreehVNuq1t5J6WXBfai9UmUluAQlNVcilKYwHMSc3E2x-YavMaTB7vsxdnmj5VJxhDL_llU4qarYpxRDbY9xd3DxZBHsGZOdMNmMyZ4xWZE9fPKkrO2aEP-S_zf9AGgvaBA</recordid><startdate>20201001</startdate><enddate>20201001</enddate><creator>Qiu, XiPeng</creator><creator>Sun, TianXiang</creator><creator>Xu, YiGe</creator><creator>Shao, YunFan</creator><creator>Dai, Ning</creator><creator>Huang, XuanJing</creator><general>Science China Press</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope></search><sort><creationdate>20201001</creationdate><title>Pre-trained models for natural language processing: A survey</title><author>Qiu, XiPeng ; Sun, TianXiang ; Xu, YiGe ; Shao, YunFan ; Dai, Ning ; Huang, XuanJing</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c359t-91787d931a77f9402479bd2e85bcf7da44d462defd558c3240547253e8969203</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2020</creationdate><topic>Engineering</topic><topic>Natural language processing</topic><topic>Review</topic><topic>Taxonomy</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Qiu, XiPeng</creatorcontrib><creatorcontrib>Sun, TianXiang</creatorcontrib><creatorcontrib>Xu, YiGe</creatorcontrib><creatorcontrib>Shao, YunFan</creatorcontrib><creatorcontrib>Dai, Ning</creatorcontrib><creatorcontrib>Huang, XuanJing</creatorcontrib><collection>CrossRef</collection><jtitle>Science China. Technological sciences</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Qiu, XiPeng</au><au>Sun, TianXiang</au><au>Xu, YiGe</au><au>Shao, YunFan</au><au>Dai, Ning</au><au>Huang, XuanJing</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Pre-trained models for natural language processing: A survey</atitle><jtitle>Science China. Technological sciences</jtitle><stitle>Sci. China Technol. Sci</stitle><date>2020-10-01</date><risdate>2020</risdate><volume>63</volume><issue>10</issue><spage>1872</spage><epage>1897</epage><pages>1872-1897</pages><issn>1674-7321</issn><eissn>1869-1900</eissn><abstract>Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.</abstract><cop>Beijing</cop><pub>Science China Press</pub><doi>10.1007/s11431-020-1647-3</doi><tpages>26</tpages></addata></record>
fulltext fulltext
identifier ISSN: 1674-7321
ispartof Science China. Technological sciences, 2020-10, Vol.63 (10), p.1872-1897
issn 1674-7321
1869-1900
language eng
recordid cdi_proquest_journals_2452108732
source Alma/SFX Local Collection; SpringerLink Journals - AutoHoldings
subjects Engineering
Natural language processing
Review
Taxonomy
title Pre-trained models for natural language processing: A survey
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-05T22%3A53%3A07IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Pre-trained%20models%20for%20natural%20language%20processing:%20A%20survey&rft.jtitle=Science%20China.%20Technological%20sciences&rft.au=Qiu,%20XiPeng&rft.date=2020-10-01&rft.volume=63&rft.issue=10&rft.spage=1872&rft.epage=1897&rft.pages=1872-1897&rft.issn=1674-7321&rft.eissn=1869-1900&rft_id=info:doi/10.1007/s11431-020-1647-3&rft_dat=%3Cproquest_cross%3E2452108732%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2452108732&rft_id=info:pmid/&rfr_iscdi=true