Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
In this study, we are interested in imbuing robots with the capability of physically-grounded task planning. Recent advancements have shown that large language models (LLMs) possess extensive knowledge useful in robotic tasks, especially in reasoning and planning. However, LLMs are constrained by th...
Gespeichert in:
Hauptverfasser: | , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | Hu, Yingdong Lin, Fanqi Zhang, Tong Yi, Li Gao, Yang |
description | In this study, we are interested in imbuing robots with the capability of
physically-grounded task planning. Recent advancements have shown that large
language models (LLMs) possess extensive knowledge useful in robotic tasks,
especially in reasoning and planning. However, LLMs are constrained by their
lack of world grounding and dependence on external affordance models to
perceive environmental information, which cannot jointly reason with LLMs. We
argue that a task planner should be an inherently grounded, unified multimodal
system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a
novel approach for long-horizon robotic planning that leverages vision-language
models (VLMs) to generate a sequence of actionable steps. ViLa directly
integrates perceptual data into its reasoning and planning process, enabling a
profound understanding of commonsense knowledge in the visual world, including
spatial layouts and object attributes. It also supports flexible multimodal
goal specification and naturally incorporates visual feedback. Our extensive
evaluation, conducted in both real-robot and simulated environments,
demonstrates ViLa's superiority over existing LLM-based planners, highlighting
its effectiveness in a wide array of open-world manipulation tasks. |
doi_str_mv | 10.48550/arxiv.2311.17842 |
format | Article |
fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2311_17842</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2311_17842</sourcerecordid><originalsourceid>FETCH-LOGICAL-a672-d29fbb20360ad1934e8e70f328a606b500db694e6354360cb4a07b25d55e1ed33</originalsourceid><addsrcrecordid>eNotj7tOw0AQRbehQIEPoMr8gM2-bdNBBAHJElFkgqis2XhsVpjdyHkAfx8TqG5z7pEOY1eCpzo3hl_j8O0PqVRCpCLLtTxnr2WMH3BHbRwI3uIeSsLNDbyEA_nehw527wSL-EUDxBbmiyrRK_ABltHFnV_Dym99DEmJodtjN6I9hjD-LthZi_2WLv93wqqH-2r2mJTP86fZbZmgzWTSyKJ1TnJlOTaiUJpyynirZI6WW2c4b5wtNFll9MisnUaeOWkaY0hQo9SETf-0p7J6M_hPHH7q38L6VKiOPKlJ4Q</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning</title><source>arXiv.org</source><creator>Hu, Yingdong ; Lin, Fanqi ; Zhang, Tong ; Yi, Li ; Gao, Yang</creator><creatorcontrib>Hu, Yingdong ; Lin, Fanqi ; Zhang, Tong ; Yi, Li ; Gao, Yang</creatorcontrib><description>In this study, we are interested in imbuing robots with the capability of
physically-grounded task planning. Recent advancements have shown that large
language models (LLMs) possess extensive knowledge useful in robotic tasks,
especially in reasoning and planning. However, LLMs are constrained by their
lack of world grounding and dependence on external affordance models to
perceive environmental information, which cannot jointly reason with LLMs. We
argue that a task planner should be an inherently grounded, unified multimodal
system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a
novel approach for long-horizon robotic planning that leverages vision-language
models (VLMs) to generate a sequence of actionable steps. ViLa directly
integrates perceptual data into its reasoning and planning process, enabling a
profound understanding of commonsense knowledge in the visual world, including
spatial layouts and object attributes. It also supports flexible multimodal
goal specification and naturally incorporates visual feedback. Our extensive
evaluation, conducted in both real-robot and simulated environments,
demonstrates ViLa's superiority over existing LLM-based planners, highlighting
its effectiveness in a wide array of open-world manipulation tasks.</description><identifier>DOI: 10.48550/arxiv.2311.17842</identifier><language>eng</language><subject>Computer Science - Artificial Intelligence ; Computer Science - Computation and Language ; Computer Science - Computer Vision and Pattern Recognition ; Computer Science - Learning ; Computer Science - Robotics</subject><creationdate>2023-11</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2311.17842$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2311.17842$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Hu, Yingdong</creatorcontrib><creatorcontrib>Lin, Fanqi</creatorcontrib><creatorcontrib>Zhang, Tong</creatorcontrib><creatorcontrib>Yi, Li</creatorcontrib><creatorcontrib>Gao, Yang</creatorcontrib><title>Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning</title><description>In this study, we are interested in imbuing robots with the capability of
physically-grounded task planning. Recent advancements have shown that large
language models (LLMs) possess extensive knowledge useful in robotic tasks,
especially in reasoning and planning. However, LLMs are constrained by their
lack of world grounding and dependence on external affordance models to
perceive environmental information, which cannot jointly reason with LLMs. We
argue that a task planner should be an inherently grounded, unified multimodal
system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a
novel approach for long-horizon robotic planning that leverages vision-language
models (VLMs) to generate a sequence of actionable steps. ViLa directly
integrates perceptual data into its reasoning and planning process, enabling a
profound understanding of commonsense knowledge in the visual world, including
spatial layouts and object attributes. It also supports flexible multimodal
goal specification and naturally incorporates visual feedback. Our extensive
evaluation, conducted in both real-robot and simulated environments,
demonstrates ViLa's superiority over existing LLM-based planners, highlighting
its effectiveness in a wide array of open-world manipulation tasks.</description><subject>Computer Science - Artificial Intelligence</subject><subject>Computer Science - Computation and Language</subject><subject>Computer Science - Computer Vision and Pattern Recognition</subject><subject>Computer Science - Learning</subject><subject>Computer Science - Robotics</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj7tOw0AQRbehQIEPoMr8gM2-bdNBBAHJElFkgqis2XhsVpjdyHkAfx8TqG5z7pEOY1eCpzo3hl_j8O0PqVRCpCLLtTxnr2WMH3BHbRwI3uIeSsLNDbyEA_nehw527wSL-EUDxBbmiyrRK_ABltHFnV_Dym99DEmJodtjN6I9hjD-LthZi_2WLv93wqqH-2r2mJTP86fZbZmgzWTSyKJ1TnJlOTaiUJpyynirZI6WW2c4b5wtNFll9MisnUaeOWkaY0hQo9SETf-0p7J6M_hPHH7q38L6VKiOPKlJ4Q</recordid><startdate>20231129</startdate><enddate>20231129</enddate><creator>Hu, Yingdong</creator><creator>Lin, Fanqi</creator><creator>Zhang, Tong</creator><creator>Yi, Li</creator><creator>Gao, Yang</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20231129</creationdate><title>Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning</title><author>Hu, Yingdong ; Lin, Fanqi ; Zhang, Tong ; Yi, Li ; Gao, Yang</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a672-d29fbb20360ad1934e8e70f328a606b500db694e6354360cb4a07b25d55e1ed33</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Computer Science - Artificial Intelligence</topic><topic>Computer Science - Computation and Language</topic><topic>Computer Science - Computer Vision and Pattern Recognition</topic><topic>Computer Science - Learning</topic><topic>Computer Science - Robotics</topic><toplevel>online_resources</toplevel><creatorcontrib>Hu, Yingdong</creatorcontrib><creatorcontrib>Lin, Fanqi</creatorcontrib><creatorcontrib>Zhang, Tong</creatorcontrib><creatorcontrib>Yi, Li</creatorcontrib><creatorcontrib>Gao, Yang</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Hu, Yingdong</au><au>Lin, Fanqi</au><au>Zhang, Tong</au><au>Yi, Li</au><au>Gao, Yang</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning</atitle><date>2023-11-29</date><risdate>2023</risdate><abstract>In this study, we are interested in imbuing robots with the capability of
physically-grounded task planning. Recent advancements have shown that large
language models (LLMs) possess extensive knowledge useful in robotic tasks,
especially in reasoning and planning. However, LLMs are constrained by their
lack of world grounding and dependence on external affordance models to
perceive environmental information, which cannot jointly reason with LLMs. We
argue that a task planner should be an inherently grounded, unified multimodal
system. To this end, we introduce Robotic Vision-Language Planning (ViLa), a
novel approach for long-horizon robotic planning that leverages vision-language
models (VLMs) to generate a sequence of actionable steps. ViLa directly
integrates perceptual data into its reasoning and planning process, enabling a
profound understanding of commonsense knowledge in the visual world, including
spatial layouts and object attributes. It also supports flexible multimodal
goal specification and naturally incorporates visual feedback. Our extensive
evaluation, conducted in both real-robot and simulated environments,
demonstrates ViLa's superiority over existing LLM-based planners, highlighting
its effectiveness in a wide array of open-world manipulation tasks.</abstract><doi>10.48550/arxiv.2311.17842</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | DOI: 10.48550/arxiv.2311.17842 |
ispartof | |
issn | |
language | eng |
recordid | cdi_arxiv_primary_2311_17842 |
source | arXiv.org |
subjects | Computer Science - Artificial Intelligence Computer Science - Computation and Language Computer Science - Computer Vision and Pattern Recognition Computer Science - Learning Computer Science - Robotics |
title | Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-05T06%3A06%3A25IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Look%20Before%20You%20Leap:%20Unveiling%20the%20Power%20of%20GPT-4V%20in%20Robotic%20Vision-Language%20Planning&rft.au=Hu,%20Yingdong&rft.date=2023-11-29&rft_id=info:doi/10.48550/arxiv.2311.17842&rft_dat=%3Carxiv_GOX%3E2311_17842%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |