Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training
Recently, we have proposed the minimum error entropy (MEE) criterion as an information theoretic alternative to the widely used mean square error criterion in supervised adaptive system training. For this purpose, we have formulated a nonparametric estimator for Renyi's entropy that employs Par...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on signal processing 2003-07, Vol.51 (7), p.1966-1978 |
|---|---|
| Hauptverfasser: | , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 1978 |
|---|---|
| container_issue | 7 |
| container_start_page | 1966 |
| container_title | IEEE transactions on signal processing |
| container_volume | 51 |
| creator | Erdogmus, D. Principe, J.C. |
| description | Recently, we have proposed the minimum error entropy (MEE) criterion as an information theoretic alternative to the widely used mean square error criterion in supervised adaptive system training. For this purpose, we have formulated a nonparametric estimator for Renyi's entropy that employs Parzen windowing. Mathematical investigation of the proposed entropy estimator revealed interesting insights about the process of information theoretical learning. This new estimator and the associated criteria have been applied to the supervised and unsupervised training of adaptive systems in a wide range of problems successfully. In this paper, we analyze the structure of the MEE performance surface around the optimal solution, and we derive the upper bound for the step size in adaptive linear neuron (ADALINE) training with the steepest descent algorithm using MEE. In addition, the effects of the entropy order and the kernel size in Parzen windowing on the shape of the performance surface and the eigenvalues of the Hessian at and around the optimal solution are investigated. Conclusions from the theoretical analyses are illustrated through numerical examples. |
| doi_str_mv | 10.1109/TSP.2003.812843 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_ieee_primary_1206704</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>1206704</ieee_id><sourcerecordid>907961303</sourcerecordid><originalsourceid>FETCH-LOGICAL-c379t-ae43151c361a7112d694d804869d19fd2b56733a2b002a814333f5aafc0f66d53</originalsourceid><addsrcrecordid>eNp90cFrFDEUBvBBFKytZw9egqCeZpuXl8kkx2VbtbDYghW8hTTzUlN2MmsyK-x_b5YtFDz0lMD7fQ8eX9O8A74A4Ob89sfNQnCOCw1CS3zRnICR0HLZq5f1zztsO93_et28KeWBc5DSqJOGVlP6S_mekie2zdOW8hypMJcGNrjZMQoh-ljHezYFNv8mNsYUx93IKOcpM0pzTe2Zz3GmHKfEYmLLi-X66vslm7OrON2fNa-C2xR6-_ieNj-_XN6uvrXr669Xq-W69dibuXUkETrwqMD1AGJQRg6aS63MACYM4q5TPaITd5wLp0EiYuicC54HpYYOT5vPx731kj87KrMdY_G02bhE065Yw3ujADlW-elZKbQQSosD_PAffJh2OdUrrNYSNGpjKjo_Ip-nUjIFu81xdHlvgdtDO7a2Yw_t2GM7NfHxca0r3m1CdsnH8hST2mCHorr3RxeJ6GksuOq5xH-3MJb3</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>884183899</pqid></control><display><type>article</type><title>Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training</title><source>IEEE Electronic Library (IEL)</source><creator>Erdogmus, D. ; Principe, J.C.</creator><creatorcontrib>Erdogmus, D. ; Principe, J.C.</creatorcontrib><description>Recently, we have proposed the minimum error entropy (MEE) criterion as an information theoretic alternative to the widely used mean square error criterion in supervised adaptive system training. For this purpose, we have formulated a nonparametric estimator for Renyi's entropy that employs Parzen windowing. Mathematical investigation of the proposed entropy estimator revealed interesting insights about the process of information theoretical learning. This new estimator and the associated criteria have been applied to the supervised and unsupervised training of adaptive systems in a wide range of problems successfully. In this paper, we analyze the structure of the MEE performance surface around the optimal solution, and we derive the upper bound for the step size in adaptive linear neuron (ADALINE) training with the steepest descent algorithm using MEE. In addition, the effects of the entropy order and the kernel size in Parzen windowing on the shape of the performance surface and the eigenvalues of the Hessian at and around the optimal solution are investigated. Conclusions from the theoretical analyses are illustrated through numerical examples.</description><identifier>ISSN: 1053-587X</identifier><identifier>EISSN: 1941-0476</identifier><identifier>DOI: 10.1109/TSP.2003.812843</identifier><identifier>CODEN: ITPRED</identifier><language>eng</language><publisher>New York, NY: IEEE</publisher><subject>Adaptive systems ; Algorithm design and analysis ; Algorithms ; Applied sciences ; Convergence ; Criteria ; Entropy ; Errors ; Estimators ; Exact sciences and technology ; Information theory ; Information, signal and communications theory ; Kernel ; Mathematical models ; Mean square error methods ; Neurons ; Optimization ; Performance analysis ; Shape ; Studies ; Telecommunications and information theory ; Training ; Upper bound</subject><ispartof>IEEE transactions on signal processing, 2003-07, Vol.51 (7), p.1966-1978</ispartof><rights>2003 INIST-CNRS</rights><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2003</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c379t-ae43151c361a7112d694d804869d19fd2b56733a2b002a814333f5aafc0f66d53</citedby><cites>FETCH-LOGICAL-c379t-ae43151c361a7112d694d804869d19fd2b56733a2b002a814333f5aafc0f66d53</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/1206704$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,777,781,793,27905,27906,54739</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/1206704$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttp://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=14893532$$DView record in Pascal Francis$$Hfree_for_read</backlink></links><search><creatorcontrib>Erdogmus, D.</creatorcontrib><creatorcontrib>Principe, J.C.</creatorcontrib><title>Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training</title><title>IEEE transactions on signal processing</title><addtitle>TSP</addtitle><description>Recently, we have proposed the minimum error entropy (MEE) criterion as an information theoretic alternative to the widely used mean square error criterion in supervised adaptive system training. For this purpose, we have formulated a nonparametric estimator for Renyi's entropy that employs Parzen windowing. Mathematical investigation of the proposed entropy estimator revealed interesting insights about the process of information theoretical learning. This new estimator and the associated criteria have been applied to the supervised and unsupervised training of adaptive systems in a wide range of problems successfully. In this paper, we analyze the structure of the MEE performance surface around the optimal solution, and we derive the upper bound for the step size in adaptive linear neuron (ADALINE) training with the steepest descent algorithm using MEE. In addition, the effects of the entropy order and the kernel size in Parzen windowing on the shape of the performance surface and the eigenvalues of the Hessian at and around the optimal solution are investigated. Conclusions from the theoretical analyses are illustrated through numerical examples.</description><subject>Adaptive systems</subject><subject>Algorithm design and analysis</subject><subject>Algorithms</subject><subject>Applied sciences</subject><subject>Convergence</subject><subject>Criteria</subject><subject>Entropy</subject><subject>Errors</subject><subject>Estimators</subject><subject>Exact sciences and technology</subject><subject>Information theory</subject><subject>Information, signal and communications theory</subject><subject>Kernel</subject><subject>Mathematical models</subject><subject>Mean square error methods</subject><subject>Neurons</subject><subject>Optimization</subject><subject>Performance analysis</subject><subject>Shape</subject><subject>Studies</subject><subject>Telecommunications and information theory</subject><subject>Training</subject><subject>Upper bound</subject><issn>1053-587X</issn><issn>1941-0476</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2003</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNp90cFrFDEUBvBBFKytZw9egqCeZpuXl8kkx2VbtbDYghW8hTTzUlN2MmsyK-x_b5YtFDz0lMD7fQ8eX9O8A74A4Ob89sfNQnCOCw1CS3zRnICR0HLZq5f1zztsO93_et28KeWBc5DSqJOGVlP6S_mekie2zdOW8hypMJcGNrjZMQoh-ljHezYFNv8mNsYUx93IKOcpM0pzTe2Zz3GmHKfEYmLLi-X66vslm7OrON2fNa-C2xR6-_ieNj-_XN6uvrXr669Xq-W69dibuXUkETrwqMD1AGJQRg6aS63MACYM4q5TPaITd5wLp0EiYuicC54HpYYOT5vPx731kj87KrMdY_G02bhE065Yw3ujADlW-elZKbQQSosD_PAffJh2OdUrrNYSNGpjKjo_Ip-nUjIFu81xdHlvgdtDO7a2Yw_t2GM7NfHxca0r3m1CdsnH8hST2mCHorr3RxeJ6GksuOq5xH-3MJb3</recordid><startdate>20030701</startdate><enddate>20030701</enddate><creator>Erdogmus, D.</creator><creator>Principe, J.C.</creator><general>IEEE</general><general>Institute of Electrical and Electronics Engineers</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>RIA</scope><scope>RIE</scope><scope>IQODW</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>F28</scope><scope>FR3</scope></search><sort><creationdate>20030701</creationdate><title>Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training</title><author>Erdogmus, D. ; Principe, J.C.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c379t-ae43151c361a7112d694d804869d19fd2b56733a2b002a814333f5aafc0f66d53</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2003</creationdate><topic>Adaptive systems</topic><topic>Algorithm design and analysis</topic><topic>Algorithms</topic><topic>Applied sciences</topic><topic>Convergence</topic><topic>Criteria</topic><topic>Entropy</topic><topic>Errors</topic><topic>Estimators</topic><topic>Exact sciences and technology</topic><topic>Information theory</topic><topic>Information, signal and communications theory</topic><topic>Kernel</topic><topic>Mathematical models</topic><topic>Mean square error methods</topic><topic>Neurons</topic><topic>Optimization</topic><topic>Performance analysis</topic><topic>Shape</topic><topic>Studies</topic><topic>Telecommunications and information theory</topic><topic>Training</topic><topic>Upper bound</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Erdogmus, D.</creatorcontrib><creatorcontrib>Principe, J.C.</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>Pascal-Francis</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>ANTE: Abstracts in New Technology & Engineering</collection><collection>Engineering Research Database</collection><jtitle>IEEE transactions on signal processing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Erdogmus, D.</au><au>Principe, J.C.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training</atitle><jtitle>IEEE transactions on signal processing</jtitle><stitle>TSP</stitle><date>2003-07-01</date><risdate>2003</risdate><volume>51</volume><issue>7</issue><spage>1966</spage><epage>1978</epage><pages>1966-1978</pages><issn>1053-587X</issn><eissn>1941-0476</eissn><coden>ITPRED</coden><abstract>Recently, we have proposed the minimum error entropy (MEE) criterion as an information theoretic alternative to the widely used mean square error criterion in supervised adaptive system training. For this purpose, we have formulated a nonparametric estimator for Renyi's entropy that employs Parzen windowing. Mathematical investigation of the proposed entropy estimator revealed interesting insights about the process of information theoretical learning. This new estimator and the associated criteria have been applied to the supervised and unsupervised training of adaptive systems in a wide range of problems successfully. In this paper, we analyze the structure of the MEE performance surface around the optimal solution, and we derive the upper bound for the step size in adaptive linear neuron (ADALINE) training with the steepest descent algorithm using MEE. In addition, the effects of the entropy order and the kernel size in Parzen windowing on the shape of the performance surface and the eigenvalues of the Hessian at and around the optimal solution are investigated. Conclusions from the theoretical analyses are illustrated through numerical examples.</abstract><cop>New York, NY</cop><pub>IEEE</pub><doi>10.1109/TSP.2003.812843</doi><tpages>13</tpages></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 1053-587X |
| ispartof | IEEE transactions on signal processing, 2003-07, Vol.51 (7), p.1966-1978 |
| issn | 1053-587X 1941-0476 |
| language | eng |
| recordid | cdi_ieee_primary_1206704 |
| source | IEEE Electronic Library (IEL) |
| subjects | Adaptive systems Algorithm design and analysis Algorithms Applied sciences Convergence Criteria Entropy Errors Estimators Exact sciences and technology Information theory Information, signal and communications theory Kernel Mathematical models Mean square error methods Neurons Optimization Performance analysis Shape Studies Telecommunications and information theory Training Upper bound |
| title | Convergence properties and data efficiency of the minimum error entropy criterion in ADALINE training |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-19T12%3A29%3A01IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Convergence%20properties%20and%20data%20efficiency%20of%20the%20minimum%20error%20entropy%20criterion%20in%20ADALINE%20training&rft.jtitle=IEEE%20transactions%20on%20signal%20processing&rft.au=Erdogmus,%20D.&rft.date=2003-07-01&rft.volume=51&rft.issue=7&rft.spage=1966&rft.epage=1978&rft.pages=1966-1978&rft.issn=1053-587X&rft.eissn=1941-0476&rft.coden=ITPRED&rft_id=info:doi/10.1109/TSP.2003.812843&rft_dat=%3Cproquest_RIE%3E907961303%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=884183899&rft_id=info:pmid/&rft_ieee_id=1206704&rfr_iscdi=true |