Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution

Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Seyde, Tim, Werner, Peter, Schwarting, Wilko, Wulfmeier, Markus, Rus, Daniela
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Seyde, Tim
Werner, Peter
Schwarting, Wilko
Wulfmeier, Markus
Rus, Daniela
description Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
doi_str_mv 10.48550/arxiv.2404.04253
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2404_04253</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2404_04253</sourcerecordid><originalsourceid>FETCH-LOGICAL-a673-668bab93d8d39114aa06ecf9b34d99532549ea180bbaa50e7c77e0156c7265203</originalsourceid><addsrcrecordid>eNo9j81KxDAYRbNxIaMP4Mq8QGvS_DXuhqKjMIyodV2-tBkNU5shSVt9e5kqru7lHrhwELqiJOelEOQGwpeb8oITnhNeCHaO3jbBz254x8_ZzqbZh0O8xa--n05b5YfkhtGPcanB97iGeIh4dukDrzs4JjfZf_Zio-_H5Pxwgc720Ed7-ZcrVN_f1dVDtn3aPFbrbQZSsUzK0oDRrCs7pinlAETadq8N453WghWCawu0JMYACGJVq5QlVMhWFVIUhK3Q9e_t4tUcg_uE8N2c_JrFj_0AOr9LfA</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</title><source>arXiv.org</source><creator>Seyde, Tim ; Werner, Peter ; Schwarting, Wilko ; Wulfmeier, Markus ; Rus, Daniela</creator><creatorcontrib>Seyde, Tim ; Werner, Peter ; Schwarting, Wilko ; Wulfmeier, Markus ; Rus, Daniela</creatorcontrib><description>Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.</description><identifier>DOI: 10.48550/arxiv.2404.04253</identifier><language>eng</language><subject>Computer Science - Artificial Intelligence ; Computer Science - Learning ; Computer Science - Robotics</subject><creationdate>2024-04</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,778,883</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2404.04253$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2404.04253$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Seyde, Tim</creatorcontrib><creatorcontrib>Werner, Peter</creatorcontrib><creatorcontrib>Schwarting, Wilko</creatorcontrib><creatorcontrib>Wulfmeier, Markus</creatorcontrib><creatorcontrib>Rus, Daniela</creatorcontrib><title>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</title><description>Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.</description><subject>Computer Science - Artificial Intelligence</subject><subject>Computer Science - Learning</subject><subject>Computer Science - Robotics</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNo9j81KxDAYRbNxIaMP4Mq8QGvS_DXuhqKjMIyodV2-tBkNU5shSVt9e5kqru7lHrhwELqiJOelEOQGwpeb8oITnhNeCHaO3jbBz254x8_ZzqbZh0O8xa--n05b5YfkhtGPcanB97iGeIh4dukDrzs4JjfZf_Zio-_H5Pxwgc720Ed7-ZcrVN_f1dVDtn3aPFbrbQZSsUzK0oDRrCs7pinlAETadq8N453WghWCawu0JMYACGJVq5QlVMhWFVIUhK3Q9e_t4tUcg_uE8N2c_JrFj_0AOr9LfA</recordid><startdate>20240405</startdate><enddate>20240405</enddate><creator>Seyde, Tim</creator><creator>Werner, Peter</creator><creator>Schwarting, Wilko</creator><creator>Wulfmeier, Markus</creator><creator>Rus, Daniela</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20240405</creationdate><title>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</title><author>Seyde, Tim ; Werner, Peter ; Schwarting, Wilko ; Wulfmeier, Markus ; Rus, Daniela</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a673-668bab93d8d39114aa06ecf9b34d99532549ea180bbaa50e7c77e0156c7265203</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Artificial Intelligence</topic><topic>Computer Science - Learning</topic><topic>Computer Science - Robotics</topic><toplevel>online_resources</toplevel><creatorcontrib>Seyde, Tim</creatorcontrib><creatorcontrib>Werner, Peter</creatorcontrib><creatorcontrib>Schwarting, Wilko</creatorcontrib><creatorcontrib>Wulfmeier, Markus</creatorcontrib><creatorcontrib>Rus, Daniela</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Seyde, Tim</au><au>Werner, Peter</au><au>Schwarting, Wilko</au><au>Wulfmeier, Markus</au><au>Rus, Daniela</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</atitle><date>2024-04-05</date><risdate>2024</risdate><abstract>Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer in the absence of action penalization in line with optimal control theory. In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency, but action costs can be detrimental to exploration during early training. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution, taking advantage of recent results in decoupled Q-learning to scale our approach to high-dimensional action spaces up to dim(A) = 38. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.</abstract><doi>10.48550/arxiv.2404.04253</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2404.04253
ispartof
issn
language eng
recordid cdi_arxiv_primary_2404_04253
source arXiv.org
subjects Computer Science - Artificial Intelligence
Computer Science - Learning
Computer Science - Robotics
title Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-16T10%3A12%3A49IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Growing%20Q-Networks:%20Solving%20Continuous%20Control%20Tasks%20with%20Adaptive%20Control%20Resolution&rft.au=Seyde,%20Tim&rft.date=2024-04-05&rft_id=info:doi/10.48550/arxiv.2404.04253&rft_dat=%3Carxiv_GOX%3E2404_04253%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true