Convolutional Neural Network With Developmental Memory for Continual Learning
Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amoun...
Gespeichert in:
Veröffentlicht in: | IEEE transaction on neural networks and learning systems 2021-06, Vol.32 (6), p.2691-2705 |
---|---|
Hauptverfasser: | , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 2705 |
---|---|
container_issue | 6 |
container_start_page | 2691 |
container_title | IEEE transaction on neural networks and learning systems |
container_volume | 32 |
creator | Park, Gyeong-Moon Yoo, Sahng-Min Kim, Jong-Hwan |
description | Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amount of previously learned tasks while adapting to novel tasks. To overcome this main barrier to continual learning with CNNs, we introduce developmental memory (DM) into a CNN, continually generating submemory networks to learn important features of individual tasks. A novel training method, referred to here as guided learning (GL), guides the newly generated submemory to become an expert on the new task, eventually improving the performance of the overall network. At the same time, the existing submemories attempt to preserve the knowledge of old tasks. Experiments on image classification tasks show that compared with the state-of-the-art algorithms, the proposed CNN with DM not only improves the classification performance on the new image task but also leads to less forgetting of previous image tasks to facilitate continual learning. |
doi_str_mv | 10.1109/TNNLS.2020.3007548 |
format | Article |
fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_ieee_primary_9145832</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9145832</ieee_id><sourcerecordid>2536030714</sourcerecordid><originalsourceid>FETCH-LOGICAL-c328t-42ca4756315cb295228d73424611daad975e248bc71fe22410adf510ff2d56c33</originalsourceid><addsrcrecordid>eNpdkE1PwzAMhiMEYtPYH4BLJS5cOhLno-kRjU9pGweG4BZlbQodbTOSdmj_nmxDO-CLLft5bflF6JzgESE4vZ7PZpOXEWDAI4pxwpk8Qn0gAmKgUh4f6uS9h4beL3EIgblg6SnqURApCMn7aDq2zdpWXVvaRlfRzHRul9of676it7L9jG7N2lR2VZumDaOpqa3bRIV1UZC2ZdOF5sRo15TNxxk6KXTlzfAvD9Dr_d18_BhPnh-exjeTOKMg25hBplnCBSU8W0DKAWSeUAZMEJJrnacJN8DkIktIYQAYwTovOMFFATkXGaUDdLXfu3L2uzO-VXXpM1NVujG28woYCCJ5uBHQy3_o0nYu_BooTgWmOCEsULCnMme9d6ZQK1fW2m0UwWrrt9r5rbZ-qz-_g-hiLyqNMQdBShiXFOgvB6l5CA</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2536030714</pqid></control><display><type>article</type><title>Convolutional Neural Network With Developmental Memory for Continual Learning</title><source>IEEE Electronic Library (IEL)</source><creator>Park, Gyeong-Moon ; Yoo, Sahng-Min ; Kim, Jong-Hwan</creator><creatorcontrib>Park, Gyeong-Moon ; Yoo, Sahng-Min ; Kim, Jong-Hwan</creatorcontrib><description>Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amount of previously learned tasks while adapting to novel tasks. To overcome this main barrier to continual learning with CNNs, we introduce developmental memory (DM) into a CNN, continually generating submemory networks to learn important features of individual tasks. A novel training method, referred to here as guided learning (GL), guides the newly generated submemory to become an expert on the new task, eventually improving the performance of the overall network. At the same time, the existing submemories attempt to preserve the knowledge of old tasks. Experiments on image classification tasks show that compared with the state-of-the-art algorithms, the proposed CNN with DM not only improves the classification performance on the new image task but also leads to less forgetting of previous image tasks to facilitate continual learning.</description><identifier>ISSN: 2162-237X</identifier><identifier>EISSN: 2162-2388</identifier><identifier>DOI: 10.1109/TNNLS.2020.3007548</identifier><identifier>PMID: 32692685</identifier><identifier>CODEN: ITNNAL</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Algorithms ; Artificial neural networks ; Biological neural networks ; Classification ; Computer vision ; Continual learning ; convolutional neural network (CNN) ; developmental memory (DM) ; Feature extraction ; guided learning (GL) ; Image classification ; Initiatives ; Knowledge engineering ; Learning ; Learning systems ; Machine learning ; Neural networks ; Task analysis ; Training ; Training data ; transfer learning</subject><ispartof>IEEE transaction on neural networks and learning systems, 2021-06, Vol.32 (6), p.2691-2705</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2021</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c328t-42ca4756315cb295228d73424611daad975e248bc71fe22410adf510ff2d56c33</citedby><cites>FETCH-LOGICAL-c328t-42ca4756315cb295228d73424611daad975e248bc71fe22410adf510ff2d56c33</cites><orcidid>0000-0003-4011-9981 ; 0000-0002-8387-3709 ; 0000-0002-4172-4174</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9145832$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/9145832$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Park, Gyeong-Moon</creatorcontrib><creatorcontrib>Yoo, Sahng-Min</creatorcontrib><creatorcontrib>Kim, Jong-Hwan</creatorcontrib><title>Convolutional Neural Network With Developmental Memory for Continual Learning</title><title>IEEE transaction on neural networks and learning systems</title><addtitle>TNNLS</addtitle><description>Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amount of previously learned tasks while adapting to novel tasks. To overcome this main barrier to continual learning with CNNs, we introduce developmental memory (DM) into a CNN, continually generating submemory networks to learn important features of individual tasks. A novel training method, referred to here as guided learning (GL), guides the newly generated submemory to become an expert on the new task, eventually improving the performance of the overall network. At the same time, the existing submemories attempt to preserve the knowledge of old tasks. Experiments on image classification tasks show that compared with the state-of-the-art algorithms, the proposed CNN with DM not only improves the classification performance on the new image task but also leads to less forgetting of previous image tasks to facilitate continual learning.</description><subject>Algorithms</subject><subject>Artificial neural networks</subject><subject>Biological neural networks</subject><subject>Classification</subject><subject>Computer vision</subject><subject>Continual learning</subject><subject>convolutional neural network (CNN)</subject><subject>developmental memory (DM)</subject><subject>Feature extraction</subject><subject>guided learning (GL)</subject><subject>Image classification</subject><subject>Initiatives</subject><subject>Knowledge engineering</subject><subject>Learning</subject><subject>Learning systems</subject><subject>Machine learning</subject><subject>Neural networks</subject><subject>Task analysis</subject><subject>Training</subject><subject>Training data</subject><subject>transfer learning</subject><issn>2162-237X</issn><issn>2162-2388</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpdkE1PwzAMhiMEYtPYH4BLJS5cOhLno-kRjU9pGweG4BZlbQodbTOSdmj_nmxDO-CLLft5bflF6JzgESE4vZ7PZpOXEWDAI4pxwpk8Qn0gAmKgUh4f6uS9h4beL3EIgblg6SnqURApCMn7aDq2zdpWXVvaRlfRzHRul9of676it7L9jG7N2lR2VZumDaOpqa3bRIV1UZC2ZdOF5sRo15TNxxk6KXTlzfAvD9Dr_d18_BhPnh-exjeTOKMg25hBplnCBSU8W0DKAWSeUAZMEJJrnacJN8DkIktIYQAYwTovOMFFATkXGaUDdLXfu3L2uzO-VXXpM1NVujG28woYCCJ5uBHQy3_o0nYu_BooTgWmOCEsULCnMme9d6ZQK1fW2m0UwWrrt9r5rbZ-qz-_g-hiLyqNMQdBShiXFOgvB6l5CA</recordid><startdate>20210601</startdate><enddate>20210601</enddate><creator>Park, Gyeong-Moon</creator><creator>Yoo, Sahng-Min</creator><creator>Kim, Jong-Hwan</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QP</scope><scope>7QQ</scope><scope>7QR</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TK</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>JG9</scope><scope>JQ2</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>P64</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0003-4011-9981</orcidid><orcidid>https://orcid.org/0000-0002-8387-3709</orcidid><orcidid>https://orcid.org/0000-0002-4172-4174</orcidid></search><sort><creationdate>20210601</creationdate><title>Convolutional Neural Network With Developmental Memory for Continual Learning</title><author>Park, Gyeong-Moon ; Yoo, Sahng-Min ; Kim, Jong-Hwan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c328t-42ca4756315cb295228d73424611daad975e248bc71fe22410adf510ff2d56c33</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Algorithms</topic><topic>Artificial neural networks</topic><topic>Biological neural networks</topic><topic>Classification</topic><topic>Computer vision</topic><topic>Continual learning</topic><topic>convolutional neural network (CNN)</topic><topic>developmental memory (DM)</topic><topic>Feature extraction</topic><topic>guided learning (GL)</topic><topic>Image classification</topic><topic>Initiatives</topic><topic>Knowledge engineering</topic><topic>Learning</topic><topic>Learning systems</topic><topic>Machine learning</topic><topic>Neural networks</topic><topic>Task analysis</topic><topic>Training</topic><topic>Training data</topic><topic>transfer learning</topic><toplevel>online_resources</toplevel><creatorcontrib>Park, Gyeong-Moon</creatorcontrib><creatorcontrib>Yoo, Sahng-Min</creatorcontrib><creatorcontrib>Kim, Jong-Hwan</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Calcium & Calcified Tissue Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Chemoreception Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical & Transportation Engineering Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology & Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transaction on neural networks and learning systems</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Park, Gyeong-Moon</au><au>Yoo, Sahng-Min</au><au>Kim, Jong-Hwan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Convolutional Neural Network With Developmental Memory for Continual Learning</atitle><jtitle>IEEE transaction on neural networks and learning systems</jtitle><stitle>TNNLS</stitle><date>2021-06-01</date><risdate>2021</risdate><volume>32</volume><issue>6</issue><spage>2691</spage><epage>2705</epage><pages>2691-2705</pages><issn>2162-237X</issn><eissn>2162-2388</eissn><coden>ITNNAL</coden><abstract>Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amount of previously learned tasks while adapting to novel tasks. To overcome this main barrier to continual learning with CNNs, we introduce developmental memory (DM) into a CNN, continually generating submemory networks to learn important features of individual tasks. A novel training method, referred to here as guided learning (GL), guides the newly generated submemory to become an expert on the new task, eventually improving the performance of the overall network. At the same time, the existing submemories attempt to preserve the knowledge of old tasks. Experiments on image classification tasks show that compared with the state-of-the-art algorithms, the proposed CNN with DM not only improves the classification performance on the new image task but also leads to less forgetting of previous image tasks to facilitate continual learning.</abstract><cop>Piscataway</cop><pub>IEEE</pub><pmid>32692685</pmid><doi>10.1109/TNNLS.2020.3007548</doi><tpages>15</tpages><orcidid>https://orcid.org/0000-0003-4011-9981</orcidid><orcidid>https://orcid.org/0000-0002-8387-3709</orcidid><orcidid>https://orcid.org/0000-0002-4172-4174</orcidid></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | ISSN: 2162-237X |
ispartof | IEEE transaction on neural networks and learning systems, 2021-06, Vol.32 (6), p.2691-2705 |
issn | 2162-237X 2162-2388 |
language | eng |
recordid | cdi_ieee_primary_9145832 |
source | IEEE Electronic Library (IEL) |
subjects | Algorithms Artificial neural networks Biological neural networks Classification Computer vision Continual learning convolutional neural network (CNN) developmental memory (DM) Feature extraction guided learning (GL) Image classification Initiatives Knowledge engineering Learning Learning systems Machine learning Neural networks Task analysis Training Training data transfer learning |
title | Convolutional Neural Network With Developmental Memory for Continual Learning |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T05%3A58%3A26IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Convolutional%20Neural%20Network%20With%20Developmental%20Memory%20for%20Continual%20Learning&rft.jtitle=IEEE%20transaction%20on%20neural%20networks%20and%20learning%20systems&rft.au=Park,%20Gyeong-Moon&rft.date=2021-06-01&rft.volume=32&rft.issue=6&rft.spage=2691&rft.epage=2705&rft.pages=2691-2705&rft.issn=2162-237X&rft.eissn=2162-2388&rft.coden=ITNNAL&rft_id=info:doi/10.1109/TNNLS.2020.3007548&rft_dat=%3Cproquest_RIE%3E2536030714%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2536030714&rft_id=info:pmid/32692685&rft_ieee_id=9145832&rfr_iscdi=true |