Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning

In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE robotics and automation letters 2019-07, Vol.4 (3), p.2282-2288
Hauptverfasser: Suarez-Hernandez, Alejandro, Torras, Carme, Alenya, Guillem
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 2288
container_issue 3
container_start_page 2282
container_title IEEE robotics and automation letters
container_volume 4
creator Suarez-Hernandez, Alejandro
Torras, Carme
Alenya, Guillem
description In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.
doi_str_mv 10.1109/LRA.2019.2901905
format Article
fullrecord <record><control><sourceid>proquest_RIE</sourceid><recordid>TN_cdi_ieee_primary_8653968</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>8653968</ieee_id><sourcerecordid>2298321828</sourcerecordid><originalsourceid>FETCH-LOGICAL-c333t-2c4d1f7ae00fe294c2893827af1e61cfb2af9e485a764933cfa692085f6ed26d3</originalsourceid><addsrcrecordid>eNpNkE1rAjEQhkNpoWK9F3oJ9Lw2H7vZ5ChqP0CpSIvHELOTGlk3Nlmh_vuuKKWXeefwvDPwIHRPyZBSop5my9GQEaqGTHWTFFeox3hZZrwU4vrffosGKW0JIbRgJVdFD60W0djWW1PjJaRQH1ofGjyHdhOqhF2IeD5ZJOwbvAzr0IEJT39gt6-981DhlW83eOKTSQl26_qIF7VpGt983aEbZ-oEg0v20efz9GP8ms3eX97Go1lmOedtxmxeUVcaIMQBU7llUnHJSuMoCGrdmhmnIJeFKUWuOLfOCMWILJyAiomK99Hj-e4-hu8DpFZvwyE23UvNmJKcUclkR5EzZWNIKYLT--h3Jh41JfpkUHcG9cmgvhjsKg_nigeAP1yKgish-S-3bWxB</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2298321828</pqid></control><display><type>article</type><title>Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning</title><source>IEEE Electronic Library (IEL)</source><creator>Suarez-Hernandez, Alejandro ; Torras, Carme ; Alenya, Guillem</creator><creatorcontrib>Suarez-Hernandez, Alejandro ; Torras, Carme ; Alenya, Guillem</creatorcontrib><description>In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.</description><identifier>ISSN: 2377-3766</identifier><identifier>EISSN: 2377-3766</identifier><identifier>DOI: 10.1109/LRA.2019.2901905</identifier><identifier>CODEN: IRALC6</identifier><language>eng</language><publisher>Piscataway: IEEE</publisher><subject>Decision making ; Dismantling ; Dynamic programming ; Fasteners ; Hierarchies ; hybrid logical/dynamical planning and verification ; Iterative methods ; Markov processes ; Planning ; Probabilistic logic ; Probability theory ; Real-time programming ; Robot kinematics ; Robotics ; scheduling and coordination ; Shortest path planning ; Task analysis ; task planning</subject><ispartof>IEEE robotics and automation letters, 2019-07, Vol.4 (3), p.2282-2288</ispartof><rights>Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c333t-2c4d1f7ae00fe294c2893827af1e61cfb2af9e485a764933cfa692085f6ed26d3</citedby><cites>FETCH-LOGICAL-c333t-2c4d1f7ae00fe294c2893827af1e61cfb2af9e485a764933cfa692085f6ed26d3</cites><orcidid>0000-0002-6018-154X ; 0000-0003-1611-614X ; 0000-0002-2933-398X</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/8653968$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>315,781,785,797,27929,27930,54763</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/8653968$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Suarez-Hernandez, Alejandro</creatorcontrib><creatorcontrib>Torras, Carme</creatorcontrib><creatorcontrib>Alenya, Guillem</creatorcontrib><title>Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning</title><title>IEEE robotics and automation letters</title><addtitle>LRA</addtitle><description>In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.</description><subject>Decision making</subject><subject>Dismantling</subject><subject>Dynamic programming</subject><subject>Fasteners</subject><subject>Hierarchies</subject><subject>hybrid logical/dynamical planning and verification</subject><subject>Iterative methods</subject><subject>Markov processes</subject><subject>Planning</subject><subject>Probabilistic logic</subject><subject>Probability theory</subject><subject>Real-time programming</subject><subject>Robot kinematics</subject><subject>Robotics</subject><subject>scheduling and coordination</subject><subject>Shortest path planning</subject><subject>Task analysis</subject><subject>task planning</subject><issn>2377-3766</issn><issn>2377-3766</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2019</creationdate><recordtype>article</recordtype><sourceid>RIE</sourceid><recordid>eNpNkE1rAjEQhkNpoWK9F3oJ9Lw2H7vZ5ChqP0CpSIvHELOTGlk3Nlmh_vuuKKWXeefwvDPwIHRPyZBSop5my9GQEaqGTHWTFFeox3hZZrwU4vrffosGKW0JIbRgJVdFD60W0djWW1PjJaRQH1ofGjyHdhOqhF2IeD5ZJOwbvAzr0IEJT39gt6-981DhlW83eOKTSQl26_qIF7VpGt983aEbZ-oEg0v20efz9GP8ms3eX97Go1lmOedtxmxeUVcaIMQBU7llUnHJSuMoCGrdmhmnIJeFKUWuOLfOCMWILJyAiomK99Hj-e4-hu8DpFZvwyE23UvNmJKcUclkR5EzZWNIKYLT--h3Jh41JfpkUHcG9cmgvhjsKg_nigeAP1yKgish-S-3bWxB</recordid><startdate>20190701</startdate><enddate>20190701</enddate><creator>Suarez-Hernandez, Alejandro</creator><creator>Torras, Carme</creator><creator>Alenya, Guillem</creator><general>IEEE</general><general>The Institute of Electrical and Electronics Engineers, Inc. (IEEE)</general><scope>97E</scope><scope>RIA</scope><scope>RIE</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7SC</scope><scope>7SP</scope><scope>8FD</scope><scope>JQ2</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><orcidid>https://orcid.org/0000-0002-6018-154X</orcidid><orcidid>https://orcid.org/0000-0003-1611-614X</orcidid><orcidid>https://orcid.org/0000-0002-2933-398X</orcidid></search><sort><creationdate>20190701</creationdate><title>Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning</title><author>Suarez-Hernandez, Alejandro ; Torras, Carme ; Alenya, Guillem</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c333t-2c4d1f7ae00fe294c2893827af1e61cfb2af9e485a764933cfa692085f6ed26d3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2019</creationdate><topic>Decision making</topic><topic>Dismantling</topic><topic>Dynamic programming</topic><topic>Fasteners</topic><topic>Hierarchies</topic><topic>hybrid logical/dynamical planning and verification</topic><topic>Iterative methods</topic><topic>Markov processes</topic><topic>Planning</topic><topic>Probabilistic logic</topic><topic>Probability theory</topic><topic>Real-time programming</topic><topic>Robot kinematics</topic><topic>Robotics</topic><topic>scheduling and coordination</topic><topic>Shortest path planning</topic><topic>Task analysis</topic><topic>task planning</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Suarez-Hernandez, Alejandro</creatorcontrib><creatorcontrib>Torras, Carme</creatorcontrib><creatorcontrib>Alenya, Guillem</creatorcontrib><collection>IEEE All-Society Periodicals Package (ASPP) 2005-present</collection><collection>IEEE All-Society Periodicals Package (ASPP) 1998-Present</collection><collection>IEEE Electronic Library (IEL)</collection><collection>CrossRef</collection><collection>Computer and Information Systems Abstracts</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts – Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>IEEE robotics and automation letters</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Suarez-Hernandez, Alejandro</au><au>Torras, Carme</au><au>Alenya, Guillem</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning</atitle><jtitle>IEEE robotics and automation letters</jtitle><stitle>LRA</stitle><date>2019-07-01</date><risdate>2019</risdate><volume>4</volume><issue>3</issue><spage>2282</spage><epage>2288</epage><pages>2282-2288</pages><issn>2377-3766</issn><eissn>2377-3766</eissn><coden>IRALC6</coden><abstract>In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.</abstract><cop>Piscataway</cop><pub>IEEE</pub><doi>10.1109/LRA.2019.2901905</doi><tpages>7</tpages><orcidid>https://orcid.org/0000-0002-6018-154X</orcidid><orcidid>https://orcid.org/0000-0003-1611-614X</orcidid><orcidid>https://orcid.org/0000-0002-2933-398X</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier ISSN: 2377-3766
ispartof IEEE robotics and automation letters, 2019-07, Vol.4 (3), p.2282-2288
issn 2377-3766
2377-3766
language eng
recordid cdi_ieee_primary_8653968
source IEEE Electronic Library (IEL)
subjects Decision making
Dismantling
Dynamic programming
Fasteners
Hierarchies
hybrid logical/dynamical planning and verification
Iterative methods
Markov processes
Planning
Probabilistic logic
Probability theory
Real-time programming
Robot kinematics
Robotics
scheduling and coordination
Shortest path planning
Task analysis
task planning
title Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-15T15%3A46%3A38IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_RIE&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Practical%20Resolution%20Methods%20for%20MDPs%20in%20Robotics%20Exemplified%20With%20Disassembly%20Planning&rft.jtitle=IEEE%20robotics%20and%20automation%20letters&rft.au=Suarez-Hernandez,%20Alejandro&rft.date=2019-07-01&rft.volume=4&rft.issue=3&rft.spage=2282&rft.epage=2288&rft.pages=2282-2288&rft.issn=2377-3766&rft.eissn=2377-3766&rft.coden=IRALC6&rft_id=info:doi/10.1109/LRA.2019.2901905&rft_dat=%3Cproquest_RIE%3E2298321828%3C/proquest_RIE%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2298321828&rft_id=info:pmid/&rft_ieee_id=8653968&rfr_iscdi=true