Analysis of Model Compression Using Knowledge Distillation

In the development of deep learning, several convolution neural network (CNN) models are designed to solve various tasks. However, these CNN models are complex and cumbersome to achieve state-of-the-art performance. The current CNN models remain to suffer from the problem of large models. Thus, mode...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE access 2022, Vol.10, p.85095-85105
Hauptverfasser: Hong, Yu-Wei, Leu, Jenq-Shiou, Faisal, Muhamad, Prakosa, Setya Widyawan
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 85105
container_issue
container_start_page 85095
container_title IEEE access
container_volume 10
creator Hong, Yu-Wei
Leu, Jenq-Shiou
Faisal, Muhamad
Prakosa, Setya Widyawan
description In the development of deep learning, several convolution neural network (CNN) models are designed to solve various tasks. However, these CNN models are complex and cumbersome to achieve state-of-the-art performance. The current CNN models remain to suffer from the problem of large models. Thus, model compression techniques are proposed to cope with the complex CNN models. Meanwhile, the selection of compressed model to suit the user requirement significantly contributes during the deployment process. This paper analyses two model compressions, namely the layerwise and the widthwise compression. The compression techniques are implemented in the MobileNetV1 model. Then, knowledge distillation is applied to compensate for the accuracy loss of the compressed model. We demonstrate the analysis of those compressed models from various perspectives and develop several suggestions on the trade-off between the performance and the compression rate. In addition, we also show that the feature that is learned by the compressed models using knowledge distillation has better representation compared to the vanilla model. Our experiment shows that the widthwise compression on MobileNetV1 achieves a compression rate of 42.27% and the layerwise compression achieves 32.42%, respectively. Furthermore, the improvement of the compressed models using knowledge distillation is notable for the widthwise compression with the increasing accuracy above 4.71%.
doi_str_mv 10.1109/ACCESS.2022.3197608
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2703098980</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9852455</ieee_id><doaj_id>oai_doaj_org_article_21f4b8a35e5a4968aea9d1b4f97eb881</doaj_id><sourcerecordid>2703098980</sourcerecordid><originalsourceid>FETCH-LOGICAL-c338t-6b1b4b801792912945027e6e56f250ddbf773fdcd5957b4122bcb3fe50a88ffa3</originalsourceid><addsrcrecordid>eNpNkM1OwzAQhC0EElXpE_QSiXOKf-LE5laFAhVFHErPlpOsK1dpXOxUiLfHJVWFL2vt7sysPoSmBM8IwfJhXpaL9XpGMaUzRmSRY3GFRpTkMmWc5df__rdoEsIOxydiixcj9DjvdPsTbEicSd5dA21Suv3BQwjWdckm2G6bvHXuu4VmC8mTDb1tW93H4R26MboNMDnXMdo8Lz7L13T18bIs56u0Zkz0aV6RKqsEJoWkklCZcUwLyIHnhnLcNJUpCmaauuHxoCojlFZ1xQxwrIUwRrMxWg6-jdM7dfB2r_2Pctqqv4bzW6V9b-sWFCUmRmnGgetM5kKDlk2MN7KASggSve4Hr4N3X0cIvdq5o48IgqIFZlgKKXDcYsNW7V0IHswllWB1Yq4G5urEXJ2ZR9V0UFkAuCik4DTjnP0CmKV8LA</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2703098980</pqid></control><display><type>article</type><title>Analysis of Model Compression Using Knowledge Distillation</title><source>IEEE Open Access Journals</source><source>DOAJ Directory of Open Access Journals</source><source>EZB-FREE-00999 freely available EZB journals</source><creator>Hong, Yu-Wei ; Leu, Jenq-Shiou ; Faisal, Muhamad ; Prakosa, Setya Widyawan</creator><creatorcontrib>Hong, Yu-Wei ; Leu, Jenq-Shiou ; Faisal, Muhamad ; Prakosa, Setya Widyawan</creatorcontrib><description>In the development of deep learning, several convolution neural network (CNN) models are designed to solve various tasks. However, these CNN models are complex and cumbersome to achieve state-of-the-art performance. The current CNN models remain to suffer from the problem of large models. Thus, model compression techniques are proposed to cope with the complex CNN models. Meanwhile, the selection of compressed model to suit the user requirement significantly contributes during the deployment process. This paper analyses two model compressions, namely the layerwise and the widthwise compression. The compression techniques are implemented in the MobileNetV1 model. Then, knowledge distillation is applied to compensate for the accuracy loss of the compressed model. We demonstrate the analysis of those compressed models from various perspectives and develop several suggestions on the trade-off between the performance and the compression rate. In addition, we also show that the feature that is learned by the compressed models using knowledge distillation has better representation compared to the vanilla model. Our experiment shows that the widthwise compression on MobileNetV1 achieves a compression rate of 42.27% and the layerwise compression achieves 32.42%, respectively. Furthermore, the improvement of the compressed models using knowledge distillation is notable for the widthwise compression with the increasing accuracy above 4.71%.</description><identifier>ISSN: 2169-3536</identifier><identifier>EISSN: 2169-3536</identifier><identifier>DOI: 10.1109/ACCESS.2022.3197608</identifier><identifier>CODEN: IAECCG</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Artificial neural networks ; Computational modeling ; Computer architecture ; Convolutional neural networks ; Deep learning ; Distillation ; Information filters ; knowledge distillation ; Knowledge engineering ; Knowledge representation ; model compression ; Task complexity ; Training data ; User requirements</subject><ispartof>IEEE access, 2022, Vol.10, p.85095-85105</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c338t-6b1b4b801792912945027e6e56f250ddbf773fdcd5957b4122bcb3fe50a88ffa3</citedby><cites>FETCH-LOGICAL-c338t-6b1b4b801792912945027e6e56f250ddbf773fdcd5957b4122bcb3fe50a88ffa3</cites><orcidid>0000-0001-7197-9912 ; 0000-0002-9135-5864</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9852455$$EHTML$$P50$$Gieee$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,864,2102,4024,27633,27923,27924,27925,54933</link.rule.ids></links><search><creatorcontrib>Hong, Yu-Wei</creatorcontrib><creatorcontrib>Leu, Jenq-Shiou</creatorcontrib><creatorcontrib>Faisal, Muhamad</creatorcontrib><creatorcontrib>Prakosa, Setya Widyawan</creatorcontrib><title>Analysis of Model Compression Using Knowledge Distillation</title><title>IEEE access</title><addtitle>Access</addtitle><description>In the development of deep learning, several convolution neural network (CNN) models are designed to solve various tasks. However, these CNN models are complex and cumbersome to achieve state-of-the-art performance. The current CNN models remain to suffer from the problem of large models. Thus, model compression techniques are proposed to cope with the complex CNN models. Meanwhile, the selection of compressed model to suit the user requirement significantly contributes during the deployment process. This paper analyses two model compressions, namely the layerwise and the widthwise compression. The compression techniques are implemented in the MobileNetV1 model. Then, knowledge distillation is applied to compensate for the accuracy loss of the compressed model. We demonstrate the analysis of those compressed models from various perspectives and develop several suggestions on the trade-off between the performance and the compression rate. In addition, we also show that the feature that is learned by the compressed models using knowledge distillation has better representation compared to the vanilla model. Our experiment shows that the widthwise compression on MobileNetV1 achieves a compression rate of 42.27% and the layerwise compression achieves 32.42%, respectively. Furthermore, the improvement of the compressed models using knowledge distillation is notable for the widthwise compression with the increasing accuracy above 4.71%.</description><subject>Artificial neural networks</subject><subject>Computational modeling</subject><subject>Computer architecture</subject><subject>Convolutional neural networks</subject><subject>Deep learning</subject><subject>Distillation</subject><subject>Information filters</subject><subject>knowledge distillation</subject><subject>Knowledge engineering</subject><subject>Knowledge representation</subject><subject>model compression</subject><subject>Task complexity</subject><subject>Training data</subject><subject>User requirements</subject><issn>2169-3536</issn><issn>2169-3536</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>ESBDL</sourceid><sourceid>RIE</sourceid><sourceid>DOA</sourceid><recordid>eNpNkM1OwzAQhC0EElXpE_QSiXOKf-LE5laFAhVFHErPlpOsK1dpXOxUiLfHJVWFL2vt7sysPoSmBM8IwfJhXpaL9XpGMaUzRmSRY3GFRpTkMmWc5df__rdoEsIOxydiixcj9DjvdPsTbEicSd5dA21Suv3BQwjWdckm2G6bvHXuu4VmC8mTDb1tW93H4R26MboNMDnXMdo8Lz7L13T18bIs56u0Zkz0aV6RKqsEJoWkklCZcUwLyIHnhnLcNJUpCmaauuHxoCojlFZ1xQxwrIUwRrMxWg6-jdM7dfB2r_2Pctqqv4bzW6V9b-sWFCUmRmnGgetM5kKDlk2MN7KASggSve4Hr4N3X0cIvdq5o48IgqIFZlgKKXDcYsNW7V0IHswllWB1Yq4G5urEXJ2ZR9V0UFkAuCik4DTjnP0CmKV8LA</recordid><startdate>2022</startdate><enddate>2022</enddate><creator>Hong, Yu-Wei</creator><creator>Leu, Jenq-Shiou</creator><creator>Faisal, Muhamad</creator><creator>Prakosa, Setya Widyawan</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>ESBDL</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>7SR</scope><scope>8BQ</scope><scope>8FD</scope><scope>JG9</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>DOA</scope><orcidid>https://orcid.org/0000-0001-7197-9912</orcidid><orcidid>https://orcid.org/0000-0002-9135-5864</orcidid></search><sort><creationdate>2022</creationdate><title>Analysis of Model Compression Using Knowledge Distillation</title><author>Hong, Yu-Wei ; Leu, Jenq-Shiou ; Faisal, Muhamad ; Prakosa, Setya Widyawan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c338t-6b1b4b801792912945027e6e56f250ddbf773fdcd5957b4122bcb3fe50a88ffa3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Artificial neural networks</topic><topic>Computational modeling</topic><topic>Computer architecture</topic><topic>Convolutional neural networks</topic><topic>Deep learning</topic><topic>Distillation</topic><topic>Information filters</topic><topic>knowledge distillation</topic><topic>Knowledge engineering</topic><topic>Knowledge representation</topic><topic>model compression</topic><topic>Task complexity</topic><topic>Training data</topic><topic>User requirements</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Hong, Yu-Wei</creatorcontrib><creatorcontrib>Leu, Jenq-Shiou</creatorcontrib><creatorcontrib>Faisal, Muhamad</creatorcontrib><creatorcontrib>Prakosa, Setya Widyawan</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE Open Access Journals</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>DOAJ Directory of Open Access Journals</collection><jtitle>IEEE access</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Hong, Yu-Wei</au><au>Leu, Jenq-Shiou</au><au>Faisal, Muhamad</au><au>Prakosa, Setya Widyawan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Analysis of Model Compression Using Knowledge Distillation</atitle><jtitle>IEEE access</jtitle><stitle>Access</stitle><date>2022</date><risdate>2022</risdate><volume>10</volume><spage>85095</spage><epage>85105</epage><pages>85095-85105</pages><issn>2169-3536</issn><eissn>2169-3536</eissn><coden>IAECCG</coden><abstract>In the development of deep learning, several convolution neural network (CNN) models are designed to solve various tasks. However, these CNN models are complex and cumbersome to achieve state-of-the-art performance. The current CNN models remain to suffer from the problem of large models. Thus, model compression techniques are proposed to cope with the complex CNN models. Meanwhile, the selection of compressed model to suit the user requirement significantly contributes during the deployment process. This paper analyses two model compressions, namely the layerwise and the widthwise compression. The compression techniques are implemented in the MobileNetV1 model. Then, knowledge distillation is applied to compensate for the accuracy loss of the compressed model. We demonstrate the analysis of those compressed models from various perspectives and develop several suggestions on the trade-off between the performance and the compression rate. In addition, we also show that the feature that is learned by the compressed models using knowledge distillation has better representation compared to the vanilla model. Our experiment shows that the widthwise compression on MobileNetV1 achieves a compression rate of 42.27% and the layerwise compression achieves 32.42%, respectively. Furthermore, the improvement of the compressed models using knowledge distillation is notable for the widthwise compression with the increasing accuracy above 4.71%.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/ACCESS.2022.3197608</doi><tpages>11</tpages><orcidid>https://orcid.org/0000-0001-7197-9912</orcidid><orcidid>https://orcid.org/0000-0002-9135-5864</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 2169-3536
ispartof IEEE access, 2022, Vol.10, p.85095-85105
issn 2169-3536
2169-3536
language eng
recordid cdi_proquest_journals_2703098980
source IEEE Open Access Journals; DOAJ Directory of Open Access Journals; EZB-FREE-00999 freely available EZB journals
subjects Artificial neural networks
Computational modeling
Computer architecture
Convolutional neural networks
Deep learning
Distillation
Information filters
knowledge distillation
Knowledge engineering
Knowledge representation
model compression
Task complexity
Training data
User requirements
title Analysis of Model Compression Using Knowledge Distillation
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T18%3A00%3A22IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Analysis%20of%20Model%20Compression%20Using%20Knowledge%20Distillation&rft.jtitle=IEEE%20access&rft.au=Hong,%20Yu-Wei&rft.date=2022&rft.volume=10&rft.spage=85095&rft.epage=85105&rft.pages=85095-85105&rft.issn=2169-3536&rft.eissn=2169-3536&rft.coden=IAECCG&rft_id=info:doi/10.1109/ACCESS.2022.3197608&rft_dat=%3Cproquest_cross%3E2703098980%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2703098980&rft_id=info:pmid/&rft_ieee_id=9852455&rft_doaj_id=oai_doaj_org_article_21f4b8a35e5a4968aea9d1b4f97eb881&rfr_iscdi=true