RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Tang, Zhenggang, Sundaralingam, Balakumar, Tremblay, Jonathan, Wen, Bowen, Yuan, Ye, Tyree, Stephen, Loop, Charles, Schwing, Alexander, Birchfield, Stan
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Tang, Zhenggang
Sundaralingam, Balakumar
Tremblay, Jonathan
Wen, Bowen
Yuan, Ye
Tyree, Stephen
Loop, Charles
Schwing, Alexander
Birchfield, Stan
description We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.
doi_str_mv 10.48550/arxiv.2210.11668
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2210_11668</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2210_11668</sourcerecordid><originalsourceid>FETCH-LOGICAL-a678-519bb5f8c0cb28fd99f746f73f6123dcee41e0c7e5622b18d1c4ffc6e210aec73</originalsourceid><addsrcrecordid>eNotj81KxDAcxHPxIKsP4Mm8QNYmbT561OKuwsri2ntJ0n8gEJOSZsV9e2v1NDAzDPND6I5W20ZxXj3o_O2_towtBqVCqGv0fto_kWMMF3wCm-Jc8tkWnyJODvfaBChpwh8WIszYpYy7FIKflwLZZQD8pqOfzkGXNYolp3CDrpwOM9z-6wb1u-e-eyGH4_61ezwQLaQinLbGcKdsZQ1TbmxbJxvhZO0EZfVoARoKlZXABWOGqpHaxjkrYPmuwcp6g-7_ZlemYcr-U-fL8Ms2rGz1D7DUSqY</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control</title><source>arXiv.org</source><creator>Tang, Zhenggang ; Sundaralingam, Balakumar ; Tremblay, Jonathan ; Wen, Bowen ; Yuan, Ye ; Tyree, Stephen ; Loop, Charles ; Schwing, Alexander ; Birchfield, Stan</creator><creatorcontrib>Tang, Zhenggang ; Sundaralingam, Balakumar ; Tremblay, Jonathan ; Wen, Bowen ; Yuan, Ye ; Tyree, Stephen ; Loop, Charles ; Schwing, Alexander ; Birchfield, Stan</creatorcontrib><description>We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.</description><identifier>DOI: 10.48550/arxiv.2210.11668</identifier><language>eng</language><subject>Computer Science - Computer Vision and Pattern Recognition ; Computer Science - Robotics</subject><creationdate>2022-10</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2210.11668$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2210.11668$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Tang, Zhenggang</creatorcontrib><creatorcontrib>Sundaralingam, Balakumar</creatorcontrib><creatorcontrib>Tremblay, Jonathan</creatorcontrib><creatorcontrib>Wen, Bowen</creatorcontrib><creatorcontrib>Yuan, Ye</creatorcontrib><creatorcontrib>Tyree, Stephen</creatorcontrib><creatorcontrib>Loop, Charles</creatorcontrib><creatorcontrib>Schwing, Alexander</creatorcontrib><creatorcontrib>Birchfield, Stan</creatorcontrib><title>RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control</title><description>We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.</description><subject>Computer Science - Computer Vision and Pattern Recognition</subject><subject>Computer Science - Robotics</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj81KxDAcxHPxIKsP4Mm8QNYmbT561OKuwsri2ntJ0n8gEJOSZsV9e2v1NDAzDPND6I5W20ZxXj3o_O2_towtBqVCqGv0fto_kWMMF3wCm-Jc8tkWnyJODvfaBChpwh8WIszYpYy7FIKflwLZZQD8pqOfzkGXNYolp3CDrpwOM9z-6wb1u-e-eyGH4_61ezwQLaQinLbGcKdsZQ1TbmxbJxvhZO0EZfVoARoKlZXABWOGqpHaxjkrYPmuwcp6g-7_ZlemYcr-U-fL8Ms2rGz1D7DUSqY</recordid><startdate>20221020</startdate><enddate>20221020</enddate><creator>Tang, Zhenggang</creator><creator>Sundaralingam, Balakumar</creator><creator>Tremblay, Jonathan</creator><creator>Wen, Bowen</creator><creator>Yuan, Ye</creator><creator>Tyree, Stephen</creator><creator>Loop, Charles</creator><creator>Schwing, Alexander</creator><creator>Birchfield, Stan</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20221020</creationdate><title>RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control</title><author>Tang, Zhenggang ; Sundaralingam, Balakumar ; Tremblay, Jonathan ; Wen, Bowen ; Yuan, Ye ; Tyree, Stephen ; Loop, Charles ; Schwing, Alexander ; Birchfield, Stan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a678-519bb5f8c0cb28fd99f746f73f6123dcee41e0c7e5622b18d1c4ffc6e210aec73</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Computer Science - Computer Vision and Pattern Recognition</topic><topic>Computer Science - Robotics</topic><toplevel>online_resources</toplevel><creatorcontrib>Tang, Zhenggang</creatorcontrib><creatorcontrib>Sundaralingam, Balakumar</creatorcontrib><creatorcontrib>Tremblay, Jonathan</creatorcontrib><creatorcontrib>Wen, Bowen</creatorcontrib><creatorcontrib>Yuan, Ye</creatorcontrib><creatorcontrib>Tyree, Stephen</creatorcontrib><creatorcontrib>Loop, Charles</creatorcontrib><creatorcontrib>Schwing, Alexander</creatorcontrib><creatorcontrib>Birchfield, Stan</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Tang, Zhenggang</au><au>Sundaralingam, Balakumar</au><au>Tremblay, Jonathan</au><au>Wen, Bowen</au><au>Yuan, Ye</au><au>Tyree, Stephen</au><au>Loop, Charles</au><au>Schwing, Alexander</au><au>Birchfield, Stan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control</atitle><date>2022-10-20</date><risdate>2022</risdate><abstract>We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.</abstract><doi>10.48550/arxiv.2210.11668</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2210.11668
ispartof
issn
language eng
recordid cdi_arxiv_primary_2210_11668
source arXiv.org
subjects Computer Science - Computer Vision and Pattern Recognition
Computer Science - Robotics
title RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-27T09%3A03%3A03IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=RGB-Only%20Reconstruction%20of%20Tabletop%20Scenes%20for%20Collision-Free%20Manipulator%20Control&rft.au=Tang,%20Zhenggang&rft.date=2022-10-20&rft_id=info:doi/10.48550/arxiv.2210.11668&rft_dat=%3Carxiv_GOX%3E2210_11668%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true