Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems
In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In c...
Gespeichert in:
Veröffentlicht in: | IEEE transaction on neural networks and learning systems 2024-01, Vol.35 (1), p.45-59 |
---|---|
Hauptverfasser: | , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 59 |
---|---|
container_issue | 1 |
container_start_page | 45 |
container_title | IEEE transaction on neural networks and learning systems |
container_volume | 35 |
creator | Ye, Jun Bian, Yougang Luo, Biao Hu, Manjiang Xu, Biao Ding, Rongjun |
description | In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method. |
doi_str_mv | 10.1109/TNNLS.2022.3172126 |
format | Article |
fullrecord | <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_proquest_journals_2911489181</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>9772751</ieee_id><sourcerecordid>2911489181</sourcerecordid><originalsourceid>FETCH-LOGICAL-c351t-20a0ed088ebd4edcbecba334fbc556aef8b1ada5326e9127d40c9ac7c2cbb1b83</originalsourceid><addsrcrecordid>eNpdkMFq3DAQhkVpaEKaF0ihCHrpxVtpJNnyMWyStrDdBHYDuQlJHoODbbmSfcjbV-lu91BdRjDfP8x8hFxztuKc1d_22-1mtwIGsBK8Ag7lO3IBvIQChNbvT__q-ZxcpfTC8iuZKmX9gZwLpaSUtb4gz-uQZjtjsVumqccBx5ne3D7SNkT6KzTYF_cRkT5MczfYnq7DOMfQ09DS2y75iDm57wak2zD23Yg20t1rmnFIH8lZa_uEV8d6SZ7u7_brH8Xm4fvP9c2m8ELxuQBmGTZMa3SNxMY79M4KIVvnlSotttpx21gloMSaQ9VI5mvrKw_eOe60uCRfD3OnGH4vmGYz5MWw7-2IYUkGyhKUhOwso1_-Q1_CEse8nYGac6lrrnmm4ED5GFKK2Jop5tvjq-HMvKk3f9WbN_XmqD6HPh9HL27A5hT5JzoDnw5Ah4indl1VUCku_gCejIgo</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2911489181</pqid></control><display><type>article</type><title>Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems</title><source>IEEE Electronic Library (IEL)</source><creator>Ye, Jun ; Bian, Yougang ; Luo, Biao ; Hu, Manjiang ; Xu, Biao ; Ding, Rongjun</creator><creatorcontrib>Ye, Jun ; Bian, Yougang ; Luo, Biao ; Hu, Manjiang ; Xu, Biao ; Ding, Rongjun</creatorcontrib><description>In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method.</description><identifier>ISSN: 2162-237X</identifier><identifier>EISSN: 2162-2388</identifier><identifier>DOI: 10.1109/TNNLS.2022.3172126</identifier><identifier>PMID: 35544498</identifier><identifier>CODEN: ITNNAL</identifier><language>eng</language><publisher>United States: IEEE</publisher><subject>Adaptive control ; Adaptive dynamic programming (ADP) ; Algorithms ; Approximation algorithms ; Convergence ; Cost analysis ; Cost function ; costate function ; Costs ; Data collection ; Discrete time systems ; Dynamic programming ; Estimation ; Information processing ; Iterative methods ; Mathematical models ; model-free control ; Nonlinear control ; Nonlinear systems ; Optimal control ; Supplements</subject><ispartof>IEEE transaction on neural networks and learning systems, 2024-01, Vol.35 (1), p.45-59</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024</rights><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c351t-20a0ed088ebd4edcbecba334fbc556aef8b1ada5326e9127d40c9ac7c2cbb1b83</citedby><cites>FETCH-LOGICAL-c351t-20a0ed088ebd4edcbecba334fbc556aef8b1ada5326e9127d40c9ac7c2cbb1b83</cites><orcidid>0000-0003-1247-1216 ; 0000-0002-3353-2586 ; 0000-0002-1860-7543 ; 0000-0003-2251-4478 ; 0000-0002-0346-7883 ; 0000-0002-7871-5386</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/9772751$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>314,780,784,796,27924,27925,54758</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/9772751$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/35544498$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Ye, Jun</creatorcontrib><creatorcontrib>Bian, Yougang</creatorcontrib><creatorcontrib>Luo, Biao</creatorcontrib><creatorcontrib>Hu, Manjiang</creatorcontrib><creatorcontrib>Xu, Biao</creatorcontrib><creatorcontrib>Ding, Rongjun</creatorcontrib><title>Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems</title><title>IEEE transaction on neural networks and learning systems</title><addtitle>TNNLS</addtitle><addtitle>IEEE Trans Neural Netw Learn Syst</addtitle><description>In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method.</description><subject>Adaptive control</subject><subject>Adaptive dynamic programming (ADP)</subject><subject>Algorithms</subject><subject>Approximation algorithms</subject><subject>Convergence</subject><subject>Cost analysis</subject><subject>Cost function</subject><subject>costate function</subject><subject>Costs</subject><subject>Data collection</subject><subject>Discrete time systems</subject><subject>Dynamic programming</subject><subject>Estimation</subject><subject>Information processing</subject><subject>Iterative methods</subject><subject>Mathematical models</subject><subject>model-free control</subject><subject>Nonlinear control</subject><subject>Nonlinear systems</subject><subject>Optimal control</subject><subject>Supplements</subject><issn>2162-237X</issn><issn>2162-2388</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpdkMFq3DAQhkVpaEKaF0ihCHrpxVtpJNnyMWyStrDdBHYDuQlJHoODbbmSfcjbV-lu91BdRjDfP8x8hFxztuKc1d_22-1mtwIGsBK8Ag7lO3IBvIQChNbvT__q-ZxcpfTC8iuZKmX9gZwLpaSUtb4gz-uQZjtjsVumqccBx5ne3D7SNkT6KzTYF_cRkT5MczfYnq7DOMfQ09DS2y75iDm57wak2zD23Yg20t1rmnFIH8lZa_uEV8d6SZ7u7_brH8Xm4fvP9c2m8ELxuQBmGTZMa3SNxMY79M4KIVvnlSotttpx21gloMSaQ9VI5mvrKw_eOe60uCRfD3OnGH4vmGYz5MWw7-2IYUkGyhKUhOwso1_-Q1_CEse8nYGac6lrrnmm4ED5GFKK2Jop5tvjq-HMvKk3f9WbN_XmqD6HPh9HL27A5hT5JzoDnw5Ah4indl1VUCku_gCejIgo</recordid><startdate>20240101</startdate><enddate>20240101</enddate><creator>Ye, Jun</creator><creator>Bian, Yougang</creator><creator>Luo, Biao</creator><creator>Hu, Manjiang</creator><creator>Xu, Biao</creator><creator>Ding, Rongjun</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QP</scope><scope>7QQ</scope><scope>7QR</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TK</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>JG9</scope><scope>JQ2</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>P64</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0003-1247-1216</orcidid><orcidid>https://orcid.org/0000-0002-3353-2586</orcidid><orcidid>https://orcid.org/0000-0002-1860-7543</orcidid><orcidid>https://orcid.org/0000-0003-2251-4478</orcidid><orcidid>https://orcid.org/0000-0002-0346-7883</orcidid><orcidid>https://orcid.org/0000-0002-7871-5386</orcidid></search><sort><creationdate>20240101</creationdate><title>Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems</title><author>Ye, Jun ; Bian, Yougang ; Luo, Biao ; Hu, Manjiang ; Xu, Biao ; Ding, Rongjun</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c351t-20a0ed088ebd4edcbecba334fbc556aef8b1ada5326e9127d40c9ac7c2cbb1b83</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Adaptive control</topic><topic>Adaptive dynamic programming (ADP)</topic><topic>Algorithms</topic><topic>Approximation algorithms</topic><topic>Convergence</topic><topic>Cost analysis</topic><topic>Cost function</topic><topic>costate function</topic><topic>Costs</topic><topic>Data collection</topic><topic>Discrete time systems</topic><topic>Dynamic programming</topic><topic>Estimation</topic><topic>Information processing</topic><topic>Iterative methods</topic><topic>Mathematical models</topic><topic>model-free control</topic><topic>Nonlinear control</topic><topic>Nonlinear systems</topic><topic>Optimal control</topic><topic>Supplements</topic><toplevel>online_resources</toplevel><creatorcontrib>Ye, Jun</creatorcontrib><creatorcontrib>Bian, Yougang</creatorcontrib><creatorcontrib>Luo, Biao</creatorcontrib><creatorcontrib>Hu, Manjiang</creatorcontrib><creatorcontrib>Xu, Biao</creatorcontrib><creatorcontrib>Ding, Rongjun</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Calcium & Calcified Tissue Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Chemoreception Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical & Transportation Engineering Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology & Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><jtitle>IEEE transaction on neural networks and learning systems</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Ye, Jun</au><au>Bian, Yougang</au><au>Luo, Biao</au><au>Hu, Manjiang</au><au>Xu, Biao</au><au>Ding, Rongjun</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems</atitle><jtitle>IEEE transaction on neural networks and learning systems</jtitle><stitle>TNNLS</stitle><addtitle>IEEE Trans Neural Netw Learn Syst</addtitle><date>2024-01-01</date><risdate>2024</risdate><volume>35</volume><issue>1</issue><spage>45</spage><epage>59</epage><pages>45-59</pages><issn>2162-237X</issn><eissn>2162-2388</eissn><coden>ITNNAL</coden><abstract>In this article, an adaptive dynamic programming (ADP) scheme utilizing a costate function is proposed for optimal control of unknown discrete-time nonlinear systems. The state-action data are obtained by interacting with the environment under the iterative scheme without any model information. In contrast with the traditional ADP scheme, the collected data in the proposed algorithm are generated with different policies, which improves data utilization in the learning process. In order to approximate the cost function more accurately and to achieve a better policy improvement direction in the case of insufficient data, a separate costate network is introduced to approximate the costate function under the actor-critic framework, and the costate is utilized as supplement information to estimate the cost function more precisely. Furthermore, convergence properties of the proposed algorithm are analyzed to demonstrate that the costate function plays a positive role in the convergence process of the cost function based on the alternate iteration mode of the costate function and cost function under a mild assumption. The uniformly ultimately bounded (UUB) property of all the variables is proven by using the Lyapunov approach. Finally, two numerical examples are presented to demonstrate the effectiveness and computation efficiency of the proposed method.</abstract><cop>United States</cop><pub>IEEE</pub><pmid>35544498</pmid><doi>10.1109/TNNLS.2022.3172126</doi><tpages>15</tpages><orcidid>https://orcid.org/0000-0003-1247-1216</orcidid><orcidid>https://orcid.org/0000-0002-3353-2586</orcidid><orcidid>https://orcid.org/0000-0002-1860-7543</orcidid><orcidid>https://orcid.org/0000-0003-2251-4478</orcidid><orcidid>https://orcid.org/0000-0002-0346-7883</orcidid><orcidid>https://orcid.org/0000-0002-7871-5386</orcidid></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | ISSN: 2162-237X |
ispartof | IEEE transaction on neural networks and learning systems, 2024-01, Vol.35 (1), p.45-59 |
issn | 2162-237X 2162-2388 |
language | eng |
recordid | cdi_proquest_journals_2911489181 |
source | IEEE Electronic Library (IEL) |
subjects | Adaptive control Adaptive dynamic programming (ADP) Algorithms Approximation algorithms Convergence Cost analysis Cost function costate function Costs Data collection Discrete time systems Dynamic programming Estimation Information processing Iterative methods Mathematical models model-free control Nonlinear control Nonlinear systems Optimal control Supplements |
title | Costate-Supplement ADP for Model-Free Optimal Control of Discrete-Time Nonlinear Systems |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T19%3A14%3A55IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Costate-Supplement%20ADP%20for%20Model-Free%20Optimal%20Control%20of%20Discrete-Time%20Nonlinear%20Systems&rft.jtitle=IEEE%20transaction%20on%20neural%20networks%20and%20learning%20systems&rft.au=Ye,%20Jun&rft.date=2024-01-01&rft.volume=35&rft.issue=1&rft.spage=45&rft.epage=59&rft.pages=45-59&rft.issn=2162-237X&rft.eissn=2162-2388&rft.coden=ITNNAL&rft_id=info:doi/10.1109/TNNLS.2022.3172126&rft_dat=%3Cproquest_RIE%3E2911489181%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2911489181&rft_id=info:pmid/35544498&rft_ieee_id=9772751&rfr_iscdi=true |