Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
Recent reinforcement learning approaches have shown surprisingly strong capabilities of bang-bang policies for solving continuous control benchmarks. The underlying coarse action space discretizations often yield favourable exploration characteristics while final performance does not visibly suffer...
Gespeichert in:
Hauptverfasser: | , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | Seyde, Tim Werner, Peter Schwarting, Wilko Wulfmeier, Markus Rus, Daniela |
description | Recent reinforcement learning approaches have shown surprisingly strong
capabilities of bang-bang policies for solving continuous control benchmarks.
The underlying coarse action space discretizations often yield favourable
exploration characteristics while final performance does not visibly suffer in
the absence of action penalization in line with optimal control theory. In
robotics applications, smooth control signals are commonly preferred to reduce
system wear and energy efficiency, but action costs can be detrimental to
exploration during early training. In this work, we aim to bridge this
performance gap by growing discrete action spaces from coarse to fine control
resolution, taking advantage of recent results in decoupled Q-learning to scale
our approach to high-dimensional action spaces up to dim(A) = 38. Our work
indicates that an adaptive control resolution in combination with value
decomposition yields simple critic-only algorithms that yield surprisingly
strong performance on continuous control tasks. |
doi_str_mv | 10.48550/arxiv.2404.04253 |
format | Article |
fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2404_04253</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2404_04253</sourcerecordid><originalsourceid>FETCH-LOGICAL-a673-668bab93d8d39114aa06ecf9b34d99532549ea180bbaa50e7c77e0156c7265203</originalsourceid><addsrcrecordid>eNo9j81KxDAYRbNxIaMP4Mq8QGvS_DXuhqKjMIyodV2-tBkNU5shSVt9e5kqru7lHrhwELqiJOelEOQGwpeb8oITnhNeCHaO3jbBz254x8_ZzqbZh0O8xa--n05b5YfkhtGPcanB97iGeIh4dukDrzs4JjfZf_Zio-_H5Pxwgc720Ed7-ZcrVN_f1dVDtn3aPFbrbQZSsUzK0oDRrCs7pinlAETadq8N453WghWCawu0JMYACGJVq5QlVMhWFVIUhK3Q9e_t4tUcg_uE8N2c_JrFj_0AOr9LfA</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</title><source>arXiv.org</source><creator>Seyde, Tim ; Werner, Peter ; Schwarting, Wilko ; Wulfmeier, Markus ; Rus, Daniela</creator><creatorcontrib>Seyde, Tim ; Werner, Peter ; Schwarting, Wilko ; Wulfmeier, Markus ; Rus, Daniela</creatorcontrib><description>Recent reinforcement learning approaches have shown surprisingly strong
capabilities of bang-bang policies for solving continuous control benchmarks.
The underlying coarse action space discretizations often yield favourable
exploration characteristics while final performance does not visibly suffer in
the absence of action penalization in line with optimal control theory. In
robotics applications, smooth control signals are commonly preferred to reduce
system wear and energy efficiency, but action costs can be detrimental to
exploration during early training. In this work, we aim to bridge this
performance gap by growing discrete action spaces from coarse to fine control
resolution, taking advantage of recent results in decoupled Q-learning to scale
our approach to high-dimensional action spaces up to dim(A) = 38. Our work
indicates that an adaptive control resolution in combination with value
decomposition yields simple critic-only algorithms that yield surprisingly
strong performance on continuous control tasks.</description><identifier>DOI: 10.48550/arxiv.2404.04253</identifier><language>eng</language><subject>Computer Science - Artificial Intelligence ; Computer Science - Learning ; Computer Science - Robotics</subject><creationdate>2024-04</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,778,883</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2404.04253$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2404.04253$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Seyde, Tim</creatorcontrib><creatorcontrib>Werner, Peter</creatorcontrib><creatorcontrib>Schwarting, Wilko</creatorcontrib><creatorcontrib>Wulfmeier, Markus</creatorcontrib><creatorcontrib>Rus, Daniela</creatorcontrib><title>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</title><description>Recent reinforcement learning approaches have shown surprisingly strong
capabilities of bang-bang policies for solving continuous control benchmarks.
The underlying coarse action space discretizations often yield favourable
exploration characteristics while final performance does not visibly suffer in
the absence of action penalization in line with optimal control theory. In
robotics applications, smooth control signals are commonly preferred to reduce
system wear and energy efficiency, but action costs can be detrimental to
exploration during early training. In this work, we aim to bridge this
performance gap by growing discrete action spaces from coarse to fine control
resolution, taking advantage of recent results in decoupled Q-learning to scale
our approach to high-dimensional action spaces up to dim(A) = 38. Our work
indicates that an adaptive control resolution in combination with value
decomposition yields simple critic-only algorithms that yield surprisingly
strong performance on continuous control tasks.</description><subject>Computer Science - Artificial Intelligence</subject><subject>Computer Science - Learning</subject><subject>Computer Science - Robotics</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNo9j81KxDAYRbNxIaMP4Mq8QGvS_DXuhqKjMIyodV2-tBkNU5shSVt9e5kqru7lHrhwELqiJOelEOQGwpeb8oITnhNeCHaO3jbBz254x8_ZzqbZh0O8xa--n05b5YfkhtGPcanB97iGeIh4dukDrzs4JjfZf_Zio-_H5Pxwgc720Ed7-ZcrVN_f1dVDtn3aPFbrbQZSsUzK0oDRrCs7pinlAETadq8N453WghWCawu0JMYACGJVq5QlVMhWFVIUhK3Q9e_t4tUcg_uE8N2c_JrFj_0AOr9LfA</recordid><startdate>20240405</startdate><enddate>20240405</enddate><creator>Seyde, Tim</creator><creator>Werner, Peter</creator><creator>Schwarting, Wilko</creator><creator>Wulfmeier, Markus</creator><creator>Rus, Daniela</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20240405</creationdate><title>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</title><author>Seyde, Tim ; Werner, Peter ; Schwarting, Wilko ; Wulfmeier, Markus ; Rus, Daniela</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a673-668bab93d8d39114aa06ecf9b34d99532549ea180bbaa50e7c77e0156c7265203</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Artificial Intelligence</topic><topic>Computer Science - Learning</topic><topic>Computer Science - Robotics</topic><toplevel>online_resources</toplevel><creatorcontrib>Seyde, Tim</creatorcontrib><creatorcontrib>Werner, Peter</creatorcontrib><creatorcontrib>Schwarting, Wilko</creatorcontrib><creatorcontrib>Wulfmeier, Markus</creatorcontrib><creatorcontrib>Rus, Daniela</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Seyde, Tim</au><au>Werner, Peter</au><au>Schwarting, Wilko</au><au>Wulfmeier, Markus</au><au>Rus, Daniela</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution</atitle><date>2024-04-05</date><risdate>2024</risdate><abstract>Recent reinforcement learning approaches have shown surprisingly strong
capabilities of bang-bang policies for solving continuous control benchmarks.
The underlying coarse action space discretizations often yield favourable
exploration characteristics while final performance does not visibly suffer in
the absence of action penalization in line with optimal control theory. In
robotics applications, smooth control signals are commonly preferred to reduce
system wear and energy efficiency, but action costs can be detrimental to
exploration during early training. In this work, we aim to bridge this
performance gap by growing discrete action spaces from coarse to fine control
resolution, taking advantage of recent results in decoupled Q-learning to scale
our approach to high-dimensional action spaces up to dim(A) = 38. Our work
indicates that an adaptive control resolution in combination with value
decomposition yields simple critic-only algorithms that yield surprisingly
strong performance on continuous control tasks.</abstract><doi>10.48550/arxiv.2404.04253</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | DOI: 10.48550/arxiv.2404.04253 |
ispartof | |
issn | |
language | eng |
recordid | cdi_arxiv_primary_2404_04253 |
source | arXiv.org |
subjects | Computer Science - Artificial Intelligence Computer Science - Learning Computer Science - Robotics |
title | Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-16T10%3A12%3A49IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Growing%20Q-Networks:%20Solving%20Continuous%20Control%20Tasks%20with%20Adaptive%20Control%20Resolution&rft.au=Seyde,%20Tim&rft.date=2024-04-05&rft_id=info:doi/10.48550/arxiv.2404.04253&rft_dat=%3Carxiv_GOX%3E2404_04253%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |