Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning
This study proposes a new efficient parameter tuning method for multi-agent simulation (MAS) using deep reinforcement learning. MAS is currently a useful tool for social sciences, but is hard to realize realistic simulations due to its computational burden for parameter tuning. This study proposes e...
Gespeichert in:
Veröffentlicht in: | World wide web (Bussum) 2023-09, Vol.26 (5), p.3535-3559 |
---|---|
Hauptverfasser: | , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 3559 |
---|---|
container_issue | 5 |
container_start_page | 3535 |
container_title | World wide web (Bussum) |
container_volume | 26 |
creator | Hirano, Masanori Izumi, Kiyoshi |
description | This study proposes a new efficient parameter tuning method for multi-agent simulation (MAS) using deep reinforcement learning. MAS is currently a useful tool for social sciences, but is hard to realize realistic simulations due to its computational burden for parameter tuning. This study proposes efficient parameter tuning to address this issue using deep reinforcement learning methods. To improve compatibility with the tuning task, our proposed method employs actor-critic-based deep reinforcement learning, such as deep deterministic policy gradient (DDPG) and soft actor-critic (SAC). In addition to the customized version of DDPG and SAC for our task, we also propose three additional components to stabilize the learning: an action converter (DDPG only), a redundant full neural network actor, and a seed fixer. For experimental verification, we employ a parameter tuning task in an artificial financial market simulation, comparing our proposed model, its ablations, and the Bayesian estimation-based baseline. The results demonstrate that our model outperforms the baseline in terms of tuning performance, indicating that the additional components of the proposed method are essential. Moreover, the critic of our model works effectively as a surrogate model, that is, as an approximate function of the simulation, which allows the actor to tune the parameters appropriately. We have also found that the SAC-based method exhibits the best and fastest convergence, which we assume is achieved by the high exploration capability of SAC. |
doi_str_mv | 10.1007/s11280-023-01197-5 |
format | Article |
fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2875642296</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2875642296</sourcerecordid><originalsourceid>FETCH-LOGICAL-c314t-5367fd101b191fd31dcda3c4d72257555b43121823a1cafdda98284b79cee4a83</originalsourceid><addsrcrecordid>eNp9kE1LxDAQhoMouK7-AU8Fz9FM0jTtURa_QPSi4C2kzXTp2qY1SRH_vVkrePM078DzzsBDyDmwS2BMXQUAXjLKuKAMoFJUHpAVSCUo5CAOUxZlkbJ8OyYnIewYY4WoYEXqJ5y96anD-Dn6d1qbgDabjDcDRvRZnF3ntlk7-myY-9hRs0UXs9ClzcRudNkc9oBFnDKPnUtkg8Oe6dH4ffmUHLWmD3j2O9fk9fbmZXNPH5_vHjbXj7QRkEcqRaFaCwxqqKC1AmxjjWhyqziXSkpZ5wI4lFwYaExrralKXua1qhrE3JRiTS6Wu5MfP2YMUe_G2bv0UvNSySLnvCoSxReq8WMIHls9-W4w_ksD03uXenGpk0v941LLVBJLKSTYbdH_nf6n9Q0ZkHi9</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2875642296</pqid></control><display><type>article</type><title>Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning</title><source>SpringerNature Journals</source><creator>Hirano, Masanori ; Izumi, Kiyoshi</creator><creatorcontrib>Hirano, Masanori ; Izumi, Kiyoshi</creatorcontrib><description>This study proposes a new efficient parameter tuning method for multi-agent simulation (MAS) using deep reinforcement learning. MAS is currently a useful tool for social sciences, but is hard to realize realistic simulations due to its computational burden for parameter tuning. This study proposes efficient parameter tuning to address this issue using deep reinforcement learning methods. To improve compatibility with the tuning task, our proposed method employs actor-critic-based deep reinforcement learning, such as deep deterministic policy gradient (DDPG) and soft actor-critic (SAC). In addition to the customized version of DDPG and SAC for our task, we also propose three additional components to stabilize the learning: an action converter (DDPG only), a redundant full neural network actor, and a seed fixer. For experimental verification, we employ a parameter tuning task in an artificial financial market simulation, comparing our proposed model, its ablations, and the Bayesian estimation-based baseline. The results demonstrate that our model outperforms the baseline in terms of tuning performance, indicating that the additional components of the proposed method are essential. Moreover, the critic of our model works effectively as a surrogate model, that is, as an approximate function of the simulation, which allows the actor to tune the parameters appropriately. We have also found that the SAC-based method exhibits the best and fastest convergence, which we assume is achieved by the high exploration capability of SAC.</description><identifier>ISSN: 1386-145X</identifier><identifier>EISSN: 1573-1413</identifier><identifier>DOI: 10.1007/s11280-023-01197-5</identifier><language>eng</language><publisher>New York: Springer US</publisher><subject>Ablation ; Computer Science ; Database Management ; Deep learning ; Information Systems Applications (incl.Internet) ; Mathematical models ; Multiagent systems ; Neural networks ; Operating Systems ; Parameters ; Simulation ; Special Issue on Fairness-driven User Behavioral Modelling and Analysis for Online Recommendation ; Tuning</subject><ispartof>World wide web (Bussum), 2023-09, Vol.26 (5), p.3535-3559</ispartof><rights>The Author(s) 2023</rights><rights>The Author(s) 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c314t-5367fd101b191fd31dcda3c4d72257555b43121823a1cafdda98284b79cee4a83</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://link.springer.com/content/pdf/10.1007/s11280-023-01197-5$$EPDF$$P50$$Gspringer$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://link.springer.com/10.1007/s11280-023-01197-5$$EHTML$$P50$$Gspringer$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,27924,27925,41488,42557,51319</link.rule.ids></links><search><creatorcontrib>Hirano, Masanori</creatorcontrib><creatorcontrib>Izumi, Kiyoshi</creatorcontrib><title>Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning</title><title>World wide web (Bussum)</title><addtitle>World Wide Web</addtitle><description>This study proposes a new efficient parameter tuning method for multi-agent simulation (MAS) using deep reinforcement learning. MAS is currently a useful tool for social sciences, but is hard to realize realistic simulations due to its computational burden for parameter tuning. This study proposes efficient parameter tuning to address this issue using deep reinforcement learning methods. To improve compatibility with the tuning task, our proposed method employs actor-critic-based deep reinforcement learning, such as deep deterministic policy gradient (DDPG) and soft actor-critic (SAC). In addition to the customized version of DDPG and SAC for our task, we also propose three additional components to stabilize the learning: an action converter (DDPG only), a redundant full neural network actor, and a seed fixer. For experimental verification, we employ a parameter tuning task in an artificial financial market simulation, comparing our proposed model, its ablations, and the Bayesian estimation-based baseline. The results demonstrate that our model outperforms the baseline in terms of tuning performance, indicating that the additional components of the proposed method are essential. Moreover, the critic of our model works effectively as a surrogate model, that is, as an approximate function of the simulation, which allows the actor to tune the parameters appropriately. We have also found that the SAC-based method exhibits the best and fastest convergence, which we assume is achieved by the high exploration capability of SAC.</description><subject>Ablation</subject><subject>Computer Science</subject><subject>Database Management</subject><subject>Deep learning</subject><subject>Information Systems Applications (incl.Internet)</subject><subject>Mathematical models</subject><subject>Multiagent systems</subject><subject>Neural networks</subject><subject>Operating Systems</subject><subject>Parameters</subject><subject>Simulation</subject><subject>Special Issue on Fairness-driven User Behavioral Modelling and Analysis for Online Recommendation</subject><subject>Tuning</subject><issn>1386-145X</issn><issn>1573-1413</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>C6C</sourceid><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GNUQQ</sourceid><recordid>eNp9kE1LxDAQhoMouK7-AU8Fz9FM0jTtURa_QPSi4C2kzXTp2qY1SRH_vVkrePM078DzzsBDyDmwS2BMXQUAXjLKuKAMoFJUHpAVSCUo5CAOUxZlkbJ8OyYnIewYY4WoYEXqJ5y96anD-Dn6d1qbgDabjDcDRvRZnF3ntlk7-myY-9hRs0UXs9ClzcRudNkc9oBFnDKPnUtkg8Oe6dH4ffmUHLWmD3j2O9fk9fbmZXNPH5_vHjbXj7QRkEcqRaFaCwxqqKC1AmxjjWhyqziXSkpZ5wI4lFwYaExrralKXua1qhrE3JRiTS6Wu5MfP2YMUe_G2bv0UvNSySLnvCoSxReq8WMIHls9-W4w_ksD03uXenGpk0v941LLVBJLKSTYbdH_nf6n9Q0ZkHi9</recordid><startdate>20230901</startdate><enddate>20230901</enddate><creator>Hirano, Masanori</creator><creator>Izumi, Kiyoshi</creator><general>Springer US</general><general>Springer Nature B.V</general><scope>C6C</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7SC</scope><scope>7XB</scope><scope>8AL</scope><scope>8FD</scope><scope>8FE</scope><scope>8FG</scope><scope>8FK</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>GNUQQ</scope><scope>HCIFZ</scope><scope>JQ2</scope><scope>K7-</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>M0N</scope><scope>P5Z</scope><scope>P62</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>Q9U</scope></search><sort><creationdate>20230901</creationdate><title>Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning</title><author>Hirano, Masanori ; Izumi, Kiyoshi</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c314t-5367fd101b191fd31dcda3c4d72257555b43121823a1cafdda98284b79cee4a83</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Ablation</topic><topic>Computer Science</topic><topic>Database Management</topic><topic>Deep learning</topic><topic>Information Systems Applications (incl.Internet)</topic><topic>Mathematical models</topic><topic>Multiagent systems</topic><topic>Neural networks</topic><topic>Operating Systems</topic><topic>Parameters</topic><topic>Simulation</topic><topic>Special Issue on Fairness-driven User Behavioral Modelling and Analysis for Online Recommendation</topic><topic>Tuning</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Hirano, Masanori</creatorcontrib><creatorcontrib>Izumi, Kiyoshi</creatorcontrib><collection>Springer Nature OA Free Journals</collection><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Computer and Information Systems Abstracts</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Computing Database (Alumni Edition)</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>Advanced Technologies & Aerospace Collection</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>ProQuest Central Student</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Computer Science Collection</collection><collection>Computer Science Database</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Computing Database</collection><collection>Advanced Technologies & Aerospace Database</collection><collection>ProQuest Advanced Technologies & Aerospace Collection</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central Basic</collection><jtitle>World wide web (Bussum)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Hirano, Masanori</au><au>Izumi, Kiyoshi</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning</atitle><jtitle>World wide web (Bussum)</jtitle><stitle>World Wide Web</stitle><date>2023-09-01</date><risdate>2023</risdate><volume>26</volume><issue>5</issue><spage>3535</spage><epage>3559</epage><pages>3535-3559</pages><issn>1386-145X</issn><eissn>1573-1413</eissn><abstract>This study proposes a new efficient parameter tuning method for multi-agent simulation (MAS) using deep reinforcement learning. MAS is currently a useful tool for social sciences, but is hard to realize realistic simulations due to its computational burden for parameter tuning. This study proposes efficient parameter tuning to address this issue using deep reinforcement learning methods. To improve compatibility with the tuning task, our proposed method employs actor-critic-based deep reinforcement learning, such as deep deterministic policy gradient (DDPG) and soft actor-critic (SAC). In addition to the customized version of DDPG and SAC for our task, we also propose three additional components to stabilize the learning: an action converter (DDPG only), a redundant full neural network actor, and a seed fixer. For experimental verification, we employ a parameter tuning task in an artificial financial market simulation, comparing our proposed model, its ablations, and the Bayesian estimation-based baseline. The results demonstrate that our model outperforms the baseline in terms of tuning performance, indicating that the additional components of the proposed method are essential. Moreover, the critic of our model works effectively as a surrogate model, that is, as an approximate function of the simulation, which allows the actor to tune the parameters appropriately. We have also found that the SAC-based method exhibits the best and fastest convergence, which we assume is achieved by the high exploration capability of SAC.</abstract><cop>New York</cop><pub>Springer US</pub><doi>10.1007/s11280-023-01197-5</doi><tpages>25</tpages><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | ISSN: 1386-145X |
ispartof | World wide web (Bussum), 2023-09, Vol.26 (5), p.3535-3559 |
issn | 1386-145X 1573-1413 |
language | eng |
recordid | cdi_proquest_journals_2875642296 |
source | SpringerNature Journals |
subjects | Ablation Computer Science Database Management Deep learning Information Systems Applications (incl.Internet) Mathematical models Multiagent systems Neural networks Operating Systems Parameters Simulation Special Issue on Fairness-driven User Behavioral Modelling and Analysis for Online Recommendation Tuning |
title | Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T18%3A43%3A57IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Neural-network-based%20parameter%20tuning%20for%20multi-agent%20simulation%20using%20deep%20reinforcement%20learning&rft.jtitle=World%20wide%20web%20(Bussum)&rft.au=Hirano,%20Masanori&rft.date=2023-09-01&rft.volume=26&rft.issue=5&rft.spage=3535&rft.epage=3559&rft.pages=3535-3559&rft.issn=1386-145X&rft.eissn=1573-1413&rft_id=info:doi/10.1007/s11280-023-01197-5&rft_dat=%3Cproquest_cross%3E2875642296%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2875642296&rft_id=info:pmid/&rfr_iscdi=true |