Two highly efficient second-order algorithms for training feedforward networks
We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive mom...
Gespeichert in:
| Veröffentlicht in: | IEEE transaction on neural networks and learning systems 2002-09, Vol.13 (5), p.1064-1074 |
|---|---|
| Hauptverfasser: | , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 1074 |
|---|---|
| container_issue | 5 |
| container_start_page | 1064 |
| container_title | IEEE transaction on neural networks and learning systems |
| container_volume | 13 |
| creator | Ampazis, N. Perantonis, S.J. |
| description | We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration. Simulations of large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge. |
| doi_str_mv | 10.1109/TNN.2002.1031939 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_proquest_miscellaneous_734246248</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>1031939</ieee_id><sourcerecordid>27779587</sourcerecordid><originalsourceid>FETCH-LOGICAL-c437t-f7c1d2238839967071b316cf57fe7dcc54ff8bb8657c355f5f68f7aa8367d8c53</originalsourceid><addsrcrecordid>eNqNkc1rGzEQxUVpadIk90ChLD0kp3X1PdIxhKYpBPfinsVaO7KVrleptMbkv6-MDS05tD3NMPzeG3iPkEtGZ4xR-2kxn884pXzGqGBW2FfklFnJWkqteF13KlVrOYcT8q6UR0qZVFS_JSfMcFlXeUrmi11q1nG1Hp4bDCH6iOPUFPRp7NuUe8xNN6xSjtN6U5qQcjPlLo5xXDUBsa-HXZf7ZsRpl_KPck7ehG4oeHGcZ-T73efF7X378O3L19ubh9ZLAVMbwLOec2GMsFYDBbYUTPugICD03isZglkujVbghVJBBW0CdJ0RGnrjlTgj1wffp5x-brFMbhOLx2HoRkzb4iwFa6U1_yZBSC41l6aSV38luYF96P8BAoBVBir48QX4mLZ5rME4y2m100ZXiB4gn1MpGYN7ynHT5WfHqNs_dLVlt2_ZHVuukg9H3-1yg_1vwbHWCrw_ABER__A7yH8BRLqqLQ</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>920871686</pqid></control><display><type>article</type><title>Two highly efficient second-order algorithms for training feedforward networks</title><source>IEEE Electronic Library (IEL)</source><creator>Ampazis, N. ; Perantonis, S.J.</creator><creatorcontrib>Ampazis, N. ; Perantonis, S.J.</creatorcontrib><description>We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration. Simulations of large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge.</description><identifier>ISSN: 1045-9227</identifier><identifier>ISSN: 2162-237X</identifier><identifier>EISSN: 1941-0093</identifier><identifier>EISSN: 2162-2388</identifier><identifier>DOI: 10.1109/TNN.2002.1031939</identifier><identifier>PMID: 18244504</identifier><identifier>CODEN: ITNNEP</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>Algorithms ; Artificial neural networks ; Backpropagation algorithms ; Computer simulation ; Cost function ; Feedforward ; Feedforward neural networks ; Inclusions ; Iterative methods ; Jacobian matrices ; Large-scale systems ; Least squares methods ; Multi-layer neural network ; Neural networks ; Optimization methods ; Training</subject><ispartof>IEEE transaction on neural networks and learning systems, 2002-09, Vol.13 (5), p.1064-1074</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2002</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c437t-f7c1d2238839967071b316cf57fe7dcc54ff8bb8657c355f5f68f7aa8367d8c53</citedby><cites>FETCH-LOGICAL-c437t-f7c1d2238839967071b316cf57fe7dcc54ff8bb8657c355f5f68f7aa8367d8c53</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/1031939$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/1031939$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/18244504$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Ampazis, N.</creatorcontrib><creatorcontrib>Perantonis, S.J.</creatorcontrib><title>Two highly efficient second-order algorithms for training feedforward networks</title><title>IEEE transaction on neural networks and learning systems</title><addtitle>TNN</addtitle><addtitle>IEEE Trans Neural Netw</addtitle><description>We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration. Simulations of large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge.</description><subject>Algorithms</subject><subject>Artificial neural networks</subject><subject>Backpropagation algorithms</subject><subject>Computer simulation</subject><subject>Cost function</subject><subject>Feedforward</subject><subject>Feedforward neural networks</subject><subject>Inclusions</subject><subject>Iterative methods</subject><subject>Jacobian matrices</subject><subject>Large-scale systems</subject><subject>Least squares methods</subject><subject>Multi-layer neural network</subject><subject>Neural networks</subject><subject>Optimization methods</subject><subject>Training</subject><issn>1045-9227</issn><issn>2162-237X</issn><issn>1941-0093</issn><issn>2162-2388</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2002</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNqNkc1rGzEQxUVpadIk90ChLD0kp3X1PdIxhKYpBPfinsVaO7KVrleptMbkv6-MDS05tD3NMPzeG3iPkEtGZ4xR-2kxn884pXzGqGBW2FfklFnJWkqteF13KlVrOYcT8q6UR0qZVFS_JSfMcFlXeUrmi11q1nG1Hp4bDCH6iOPUFPRp7NuUe8xNN6xSjtN6U5qQcjPlLo5xXDUBsa-HXZf7ZsRpl_KPck7ehG4oeHGcZ-T73efF7X378O3L19ubh9ZLAVMbwLOec2GMsFYDBbYUTPugICD03isZglkujVbghVJBBW0CdJ0RGnrjlTgj1wffp5x-brFMbhOLx2HoRkzb4iwFa6U1_yZBSC41l6aSV38luYF96P8BAoBVBir48QX4mLZ5rME4y2m100ZXiB4gn1MpGYN7ynHT5WfHqNs_dLVlt2_ZHVuukg9H3-1yg_1vwbHWCrw_ABER__A7yH8BRLqqLQ</recordid><startdate>20020901</startdate><enddate>20020901</enddate><creator>Ampazis, N.</creator><creator>Perantonis, S.J.</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>RIA</scope><scope>RIE</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QP</scope><scope>7QQ</scope><scope>7QR</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TK</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>JG9</scope><scope>JQ2</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>P64</scope><scope>7X8</scope></search><sort><creationdate>20020901</creationdate><title>Two highly efficient second-order algorithms for training feedforward networks</title><author>Ampazis, N. ; Perantonis, S.J.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c437t-f7c1d2238839967071b316cf57fe7dcc54ff8bb8657c355f5f68f7aa8367d8c53</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2002</creationdate><topic>Algorithms</topic><topic>Artificial neural networks</topic><topic>Backpropagation algorithms</topic><topic>Computer simulation</topic><topic>Cost function</topic><topic>Feedforward</topic><topic>Feedforward neural networks</topic><topic>Inclusions</topic><topic>Iterative methods</topic><topic>Jacobian matrices</topic><topic>Large-scale systems</topic><topic>Least squares methods</topic><topic>Multi-layer neural network</topic><topic>Neural networks</topic><topic>Optimization methods</topic><topic>Training</topic><toplevel>online_resources</toplevel><creatorcontrib>Ampazis, N.</creatorcontrib><creatorcontrib>Perantonis, S.J.</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Calcium & Calcified Tissue Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Chemoreception Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical & Transportation Engineering Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology & Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transaction on neural networks and learning systems</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Ampazis, N.</au><au>Perantonis, S.J.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Two highly efficient second-order algorithms for training feedforward networks</atitle><jtitle>IEEE transaction on neural networks and learning systems</jtitle><stitle>TNN</stitle><addtitle>IEEE Trans Neural Netw</addtitle><date>2002-09-01</date><risdate>2002</risdate><volume>13</volume><issue>5</issue><spage>1064</spage><epage>1074</epage><pages>1064-1074</pages><issn>1045-9227</issn><issn>2162-237X</issn><eissn>1941-0093</eissn><eissn>2162-2388</eissn><coden>ITNNEP</coden><abstract>We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration. Simulations of large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge.</abstract><cop>United States</cop><pub>IEEE</pub><pmid>18244504</pmid><doi>10.1109/TNN.2002.1031939</doi><tpages>11</tpages></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | ISSN: 1045-9227 |
| ispartof | IEEE transaction on neural networks and learning systems, 2002-09, Vol.13 (5), p.1064-1074 |
| issn | 1045-9227 2162-237X 1941-0093 2162-2388 |
| language | eng |
| recordid | cdi_proquest_miscellaneous_734246248 |
| source | IEEE Electronic Library (IEL) |
| subjects | Algorithms Artificial neural networks Backpropagation algorithms Computer simulation Cost function Feedforward Feedforward neural networks Inclusions Iterative methods Jacobian matrices Large-scale systems Least squares methods Multi-layer neural network Neural networks Optimization methods Training |
| title | Two highly efficient second-order algorithms for training feedforward networks |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-04T07%3A13%3A17IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Two%20highly%20efficient%20second-order%20algorithms%20for%20training%20feedforward%20networks&rft.jtitle=IEEE%20transaction%20on%20neural%20networks%20and%20learning%20systems&rft.au=Ampazis,%20N.&rft.date=2002-09-01&rft.volume=13&rft.issue=5&rft.spage=1064&rft.epage=1074&rft.pages=1064-1074&rft.issn=1045-9227&rft.eissn=1941-0093&rft.coden=ITNNEP&rft_id=info:doi/10.1109/TNN.2002.1031939&rft_dat=%3Cproquest_RIE%3E27779587%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=920871686&rft_id=info:pmid/18244504&rft_ieee_id=1031939&rfr_iscdi=true |