HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods

Activation functions are essential components in any neural network model; they play a crucial role in determining the network's expressive power through their introduced non-linearity. Rectified Linear Unit (ReLU) has been the famous and default choice for most deep neural network models becau...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE access 2023-01, Vol.11, p.1-1
Hauptverfasser: Abdel-Nabi, Heba, Al-Naymat, Ghazi, Ali, Mostafa, Awajan, Arafat
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 1
container_issue
container_start_page 1
container_title IEEE access
container_volume 11
creator Abdel-Nabi, Heba
Al-Naymat, Ghazi
Ali, Mostafa
Awajan, Arafat
description Activation functions are essential components in any neural network model; they play a crucial role in determining the network's expressive power through their introduced non-linearity. Rectified Linear Unit (ReLU) has been the famous and default choice for most deep neural network models because of its simplicity and ability to tackle the vanishing gradient problem that faces backpropagation optimization. However, ReLU introduces other challenges that hinder its performance; bias shift and dying neurons in the negative region. To address these problems, this paper introduces a novel composite monotonic, zero-centered, semi-saturated activation function called Hyperbolic cosine Linearized SquasHing function (HcLSH) with partial gradient-based sparsity HcLSH owns many desirable properties, such as considering the contribution of the negative values of neurons while having a smooth output landscape to enhance the gradient flow during training. Furthermore, the regularization effect resulting from the self-gating property of the positive region of HcLSH reduces the risk of model overfitting and ensures learning more robust expressive representations. An extensive set of experiments and comparisons is conducted that includes four popular image classification datasets, seven deep network architectures, and ten state-of-the-art activation functions. HcLSH exhibited the Top-1 and Top-3 testing accuracy results in 20 and 25 out of 28 conducted experiments, respectively, suppressing the widely used ReLU that achieved 2 and 5, and the reputable Mish that achieved 0 and 5 Top-1 and Top-3 testing accuracy results, respectively. HcLSH attained improvements over ReLU, ranging from 0.2% to 96.4% in different models and datasets. Statistical results demonstrate the significance of the enhanced performance achieved by our proposed HcLSH activation function compared to the competitive activation functions in various datasets and models regarding the testing loss Furthermore, the ablation study further verifies the proposed activation function's robustness, stability, and adaptability for the different model parameter.
doi_str_mv 10.1109/ACCESS.2023.3276298
format Article
fullrecord <record><control><sourceid>proquest_ieee_</sourceid><recordid>TN_cdi_ieee_primary_10124188</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>10124188</ieee_id><doaj_id>oai_doaj_org_article_68dccd0c0383424783f89d43499164b1</doaj_id><sourcerecordid>2818366815</sourcerecordid><originalsourceid>FETCH-LOGICAL-c409t-5fe0eeb67336cead0951ee0757cf0acf5338e28c3b089d656baec22c5ca7b6fc3</originalsourceid><addsrcrecordid>eNpNUcFq4zAQNWULDWm_YHsw7NmppLFkeW8h2zYFd8uS3bOQx-NUIStlZafQv69alyVzmHkM770ZeFn2lbMF56y-Wa5Wt5vNQjABCxCVErU-y2aCq7oACerLCb7IroZhx1LptJLVLPu1xmaz_p4v85_hhfap-6JxnmzMH4MPY_AO8yWO7sWOLvj87ujxA_Qh5j-IDnmTuN75bf5I43PohsvsvLf7ga4-5zz7c3f7e7Uumqf7h9WyKbBk9VjInhhRqyoAhWQ7VktOxCpZYc8s9hJAk9AILdN1p6RqLaEQKNFWreoR5tnD5NsFuzOH6P7a-GqCdeZjEeLW2Dg63JNRukPsGDLQUIqy0tAnzxLKuuaqbHny-jZ5HWL4d6RhNLtwjD69b4TmGpTSXCYWTCyMYRgi9f-vcmbeozBTFOY9CvMZRVJdTypHRCcKLkquNbwBNSiDgw</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2818366815</pqid></control><display><type>article</type><title>HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods</title><source>Directory of Open Access Journals</source><source>IEEE Xplore Open Access Journals</source><source>EZB Electronic Journals Library</source><creator>Abdel-Nabi, Heba ; Al-Naymat, Ghazi ; Ali, Mostafa ; Awajan, Arafat</creator><creatorcontrib>Abdel-Nabi, Heba ; Al-Naymat, Ghazi ; Ali, Mostafa ; Awajan, Arafat</creatorcontrib><description>Activation functions are essential components in any neural network model; they play a crucial role in determining the network's expressive power through their introduced non-linearity. Rectified Linear Unit (ReLU) has been the famous and default choice for most deep neural network models because of its simplicity and ability to tackle the vanishing gradient problem that faces backpropagation optimization. However, ReLU introduces other challenges that hinder its performance; bias shift and dying neurons in the negative region. To address these problems, this paper introduces a novel composite monotonic, zero-centered, semi-saturated activation function called Hyperbolic cosine Linearized SquasHing function (HcLSH) with partial gradient-based sparsity HcLSH owns many desirable properties, such as considering the contribution of the negative values of neurons while having a smooth output landscape to enhance the gradient flow during training. Furthermore, the regularization effect resulting from the self-gating property of the positive region of HcLSH reduces the risk of model overfitting and ensures learning more robust expressive representations. An extensive set of experiments and comparisons is conducted that includes four popular image classification datasets, seven deep network architectures, and ten state-of-the-art activation functions. HcLSH exhibited the Top-1 and Top-3 testing accuracy results in 20 and 25 out of 28 conducted experiments, respectively, suppressing the widely used ReLU that achieved 2 and 5, and the reputable Mish that achieved 0 and 5 Top-1 and Top-3 testing accuracy results, respectively. HcLSH attained improvements over ReLU, ranging from 0.2% to 96.4% in different models and datasets. Statistical results demonstrate the significance of the enhanced performance achieved by our proposed HcLSH activation function compared to the competitive activation functions in various datasets and models regarding the testing loss Furthermore, the ablation study further verifies the proposed activation function's robustness, stability, and adaptability for the different model parameter.</description><identifier>ISSN: 2169-3536</identifier><identifier>EISSN: 2169-3536</identifier><identifier>DOI: 10.1109/ACCESS.2023.3276298</identifier><identifier>CODEN: IAECCG</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Ablation ; Accuracy ; Activation analysis ; Activation Function ; Adaptation models ; Artificial neural networks ; Back propagation networks ; Biological neural networks ; Computer architecture ; Convergence ; Datasets ; Deep learning ; Gradient flow ; Hyperbolic functions ; Image classification ; Image Classification Accuracy ; Machine learning ; Monotonicity ; Neural networks ; Neurons ; Optimization ; Performance enhancement ; Regularization ; Saturation ; Trigonometric functions</subject><ispartof>IEEE access, 2023-01, Vol.11, p.1-1</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c409t-5fe0eeb67336cead0951ee0757cf0acf5338e28c3b089d656baec22c5ca7b6fc3</citedby><cites>FETCH-LOGICAL-c409t-5fe0eeb67336cead0951ee0757cf0acf5338e28c3b089d656baec22c5ca7b6fc3</cites><orcidid>0000-0003-3030-848X ; 0000-0002-7067-5658 ; 0000-0002-9661-5354 ; 0000-0001-6238-3244</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/10124188$$EHTML$$P50$$Gieee$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,864,2102,27633,27924,27925,54933</link.rule.ids></links><search><creatorcontrib>Abdel-Nabi, Heba</creatorcontrib><creatorcontrib>Al-Naymat, Ghazi</creatorcontrib><creatorcontrib>Ali, Mostafa</creatorcontrib><creatorcontrib>Awajan, Arafat</creatorcontrib><title>HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods</title><title>IEEE access</title><addtitle>Access</addtitle><description>Activation functions are essential components in any neural network model; they play a crucial role in determining the network's expressive power through their introduced non-linearity. Rectified Linear Unit (ReLU) has been the famous and default choice for most deep neural network models because of its simplicity and ability to tackle the vanishing gradient problem that faces backpropagation optimization. However, ReLU introduces other challenges that hinder its performance; bias shift and dying neurons in the negative region. To address these problems, this paper introduces a novel composite monotonic, zero-centered, semi-saturated activation function called Hyperbolic cosine Linearized SquasHing function (HcLSH) with partial gradient-based sparsity HcLSH owns many desirable properties, such as considering the contribution of the negative values of neurons while having a smooth output landscape to enhance the gradient flow during training. Furthermore, the regularization effect resulting from the self-gating property of the positive region of HcLSH reduces the risk of model overfitting and ensures learning more robust expressive representations. An extensive set of experiments and comparisons is conducted that includes four popular image classification datasets, seven deep network architectures, and ten state-of-the-art activation functions. HcLSH exhibited the Top-1 and Top-3 testing accuracy results in 20 and 25 out of 28 conducted experiments, respectively, suppressing the widely used ReLU that achieved 2 and 5, and the reputable Mish that achieved 0 and 5 Top-1 and Top-3 testing accuracy results, respectively. HcLSH attained improvements over ReLU, ranging from 0.2% to 96.4% in different models and datasets. Statistical results demonstrate the significance of the enhanced performance achieved by our proposed HcLSH activation function compared to the competitive activation functions in various datasets and models regarding the testing loss Furthermore, the ablation study further verifies the proposed activation function's robustness, stability, and adaptability for the different model parameter.</description><subject>Ablation</subject><subject>Accuracy</subject><subject>Activation analysis</subject><subject>Activation Function</subject><subject>Adaptation models</subject><subject>Artificial neural networks</subject><subject>Back propagation networks</subject><subject>Biological neural networks</subject><subject>Computer architecture</subject><subject>Convergence</subject><subject>Datasets</subject><subject>Deep learning</subject><subject>Gradient flow</subject><subject>Hyperbolic functions</subject><subject>Image classification</subject><subject>Image Classification Accuracy</subject><subject>Machine learning</subject><subject>Monotonicity</subject><subject>Neural networks</subject><subject>Neurons</subject><subject>Optimization</subject><subject>Performance enhancement</subject><subject>Regularization</subject><subject>Saturation</subject><subject>Trigonometric functions</subject><issn>2169-3536</issn><issn>2169-3536</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>ESBDL</sourceid><sourceid>RIE</sourceid><sourceid>DOA</sourceid><recordid>eNpNUcFq4zAQNWULDWm_YHsw7NmppLFkeW8h2zYFd8uS3bOQx-NUIStlZafQv69alyVzmHkM770ZeFn2lbMF56y-Wa5Wt5vNQjABCxCVErU-y2aCq7oACerLCb7IroZhx1LptJLVLPu1xmaz_p4v85_hhfap-6JxnmzMH4MPY_AO8yWO7sWOLvj87ujxA_Qh5j-IDnmTuN75bf5I43PohsvsvLf7ga4-5zz7c3f7e7Uumqf7h9WyKbBk9VjInhhRqyoAhWQ7VktOxCpZYc8s9hJAk9AILdN1p6RqLaEQKNFWreoR5tnD5NsFuzOH6P7a-GqCdeZjEeLW2Dg63JNRukPsGDLQUIqy0tAnzxLKuuaqbHny-jZ5HWL4d6RhNLtwjD69b4TmGpTSXCYWTCyMYRgi9f-vcmbeozBTFOY9CvMZRVJdTypHRCcKLkquNbwBNSiDgw</recordid><startdate>20230101</startdate><enddate>20230101</enddate><creator>Abdel-Nabi, Heba</creator><creator>Al-Naymat, Ghazi</creator><creator>Ali, Mostafa</creator><creator>Awajan, Arafat</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>ESBDL</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>7SR</scope><scope>8BQ</scope><scope>8FD</scope><scope>JG9</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>DOA</scope><orcidid>https://orcid.org/0000-0003-3030-848X</orcidid><orcidid>https://orcid.org/0000-0002-7067-5658</orcidid><orcidid>https://orcid.org/0000-0002-9661-5354</orcidid><orcidid>https://orcid.org/0000-0001-6238-3244</orcidid></search><sort><creationdate>20230101</creationdate><title>HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods</title><author>Abdel-Nabi, Heba ; Al-Naymat, Ghazi ; Ali, Mostafa ; Awajan, Arafat</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c409t-5fe0eeb67336cead0951ee0757cf0acf5338e28c3b089d656baec22c5ca7b6fc3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Ablation</topic><topic>Accuracy</topic><topic>Activation analysis</topic><topic>Activation Function</topic><topic>Adaptation models</topic><topic>Artificial neural networks</topic><topic>Back propagation networks</topic><topic>Biological neural networks</topic><topic>Computer architecture</topic><topic>Convergence</topic><topic>Datasets</topic><topic>Deep learning</topic><topic>Gradient flow</topic><topic>Hyperbolic functions</topic><topic>Image classification</topic><topic>Image Classification Accuracy</topic><topic>Machine learning</topic><topic>Monotonicity</topic><topic>Neural networks</topic><topic>Neurons</topic><topic>Optimization</topic><topic>Performance enhancement</topic><topic>Regularization</topic><topic>Saturation</topic><topic>Trigonometric functions</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Abdel-Nabi, Heba</creatorcontrib><creatorcontrib>Al-Naymat, Ghazi</creatorcontrib><creatorcontrib>Ali, Mostafa</creatorcontrib><creatorcontrib>Awajan, Arafat</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005–Present</collection><collection>IEEE Xplore Open Access Journals</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998–Present</collection><collection>IEEE Electronic Library Online</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Directory of Open Access Journals</collection><jtitle>IEEE access</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Abdel-Nabi, Heba</au><au>Al-Naymat, Ghazi</au><au>Ali, Mostafa</au><au>Awajan, Arafat</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods</atitle><jtitle>IEEE access</jtitle><stitle>Access</stitle><date>2023-01-01</date><risdate>2023</risdate><volume>11</volume><spage>1</spage><epage>1</epage><pages>1-1</pages><issn>2169-3536</issn><eissn>2169-3536</eissn><coden>IAECCG</coden><abstract>Activation functions are essential components in any neural network model; they play a crucial role in determining the network's expressive power through their introduced non-linearity. Rectified Linear Unit (ReLU) has been the famous and default choice for most deep neural network models because of its simplicity and ability to tackle the vanishing gradient problem that faces backpropagation optimization. However, ReLU introduces other challenges that hinder its performance; bias shift and dying neurons in the negative region. To address these problems, this paper introduces a novel composite monotonic, zero-centered, semi-saturated activation function called Hyperbolic cosine Linearized SquasHing function (HcLSH) with partial gradient-based sparsity HcLSH owns many desirable properties, such as considering the contribution of the negative values of neurons while having a smooth output landscape to enhance the gradient flow during training. Furthermore, the regularization effect resulting from the self-gating property of the positive region of HcLSH reduces the risk of model overfitting and ensures learning more robust expressive representations. An extensive set of experiments and comparisons is conducted that includes four popular image classification datasets, seven deep network architectures, and ten state-of-the-art activation functions. HcLSH exhibited the Top-1 and Top-3 testing accuracy results in 20 and 25 out of 28 conducted experiments, respectively, suppressing the widely used ReLU that achieved 2 and 5, and the reputable Mish that achieved 0 and 5 Top-1 and Top-3 testing accuracy results, respectively. HcLSH attained improvements over ReLU, ranging from 0.2% to 96.4% in different models and datasets. Statistical results demonstrate the significance of the enhanced performance achieved by our proposed HcLSH activation function compared to the competitive activation functions in various datasets and models regarding the testing loss Furthermore, the ablation study further verifies the proposed activation function's robustness, stability, and adaptability for the different model parameter.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/ACCESS.2023.3276298</doi><tpages>1</tpages><orcidid>https://orcid.org/0000-0003-3030-848X</orcidid><orcidid>https://orcid.org/0000-0002-7067-5658</orcidid><orcidid>https://orcid.org/0000-0002-9661-5354</orcidid><orcidid>https://orcid.org/0000-0001-6238-3244</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 2169-3536
ispartof IEEE access, 2023-01, Vol.11, p.1-1
issn 2169-3536
2169-3536
language eng
recordid cdi_ieee_primary_10124188
source Directory of Open Access Journals; IEEE Xplore Open Access Journals; EZB Electronic Journals Library
subjects Ablation
Accuracy
Activation analysis
Activation Function
Adaptation models
Artificial neural networks
Back propagation networks
Biological neural networks
Computer architecture
Convergence
Datasets
Deep learning
Gradient flow
Hyperbolic functions
Image classification
Image Classification Accuracy
Machine learning
Monotonicity
Neural networks
Neurons
Optimization
Performance enhancement
Regularization
Saturation
Trigonometric functions
title HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T05%3A08%3A54IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_ieee_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=HcLSH:%20A%20Novel%20Non-Linear%20Monotonic%20Activation%20Function%20for%20Deep%20Learning%20Methods&rft.jtitle=IEEE%20access&rft.au=Abdel-Nabi,%20Heba&rft.date=2023-01-01&rft.volume=11&rft.spage=1&rft.epage=1&rft.pages=1-1&rft.issn=2169-3536&rft.eissn=2169-3536&rft.coden=IAECCG&rft_id=info:doi/10.1109/ACCESS.2023.3276298&rft_dat=%3Cproquest_ieee_%3E2818366815%3C/proquest_ieee_%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2818366815&rft_id=info:pmid/&rft_ieee_id=10124188&rft_doaj_id=oai_doaj_org_article_68dccd0c0383424783f89d43499164b1&rfr_iscdi=true