Diagnosing Robotics Systems Issues with Large Language Models

Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing l...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Herrmann, Jordis Emilia, Gopinath, Aswath Mandakath, Norrlof, Mikael, Müller, Mark Niklas
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Herrmann, Jordis Emilia
Gopinath, Aswath Mandakath
Norrlof, Mikael
Müller, Mark Niklas
description Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.
doi_str_mv 10.48550/arxiv.2410.09084
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2410_09084</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2410_09084</sourcerecordid><originalsourceid>FETCH-arxiv_primary_2410_090843</originalsourceid><addsrcrecordid>eNpjYJA0NNAzsTA1NdBPLKrILNMzMgEKGFgaWJhwMti6ZCam5-UXZ-alKwTlJ-WXZCYXKwRXFpek5hYreBYXl6YWK5RnlmQo-CQWpacCybz00kQgwzc_JTWnmIeBNS0xpziVF0pzM8i7uYY4e-iCLYovKMrMTSyqjAdZGA-20JiwCgBDQDWQ</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Diagnosing Robotics Systems Issues with Large Language Models</title><source>arXiv.org</source><creator>Herrmann, Jordis Emilia ; Gopinath, Aswath Mandakath ; Norrlof, Mikael ; Müller, Mark Niklas</creator><creatorcontrib>Herrmann, Jordis Emilia ; Gopinath, Aswath Mandakath ; Norrlof, Mikael ; Müller, Mark Niklas</creatorcontrib><description>Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.</description><identifier>DOI: 10.48550/arxiv.2410.09084</identifier><language>eng</language><subject>Computer Science - Artificial Intelligence ; Computer Science - Computation and Language ; Computer Science - Learning ; Computer Science - Robotics</subject><creationdate>2024-10</creationdate><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2410.09084$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2410.09084$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Herrmann, Jordis Emilia</creatorcontrib><creatorcontrib>Gopinath, Aswath Mandakath</creatorcontrib><creatorcontrib>Norrlof, Mikael</creatorcontrib><creatorcontrib>Müller, Mark Niklas</creatorcontrib><title>Diagnosing Robotics Systems Issues with Large Language Models</title><description>Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.</description><subject>Computer Science - Artificial Intelligence</subject><subject>Computer Science - Computation and Language</subject><subject>Computer Science - Learning</subject><subject>Computer Science - Robotics</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNpjYJA0NNAzsTA1NdBPLKrILNMzMgEKGFgaWJhwMti6ZCam5-UXZ-alKwTlJ-WXZCYXKwRXFpek5hYreBYXl6YWK5RnlmQo-CQWpacCybz00kQgwzc_JTWnmIeBNS0xpziVF0pzM8i7uYY4e-iCLYovKMrMTSyqjAdZGA-20JiwCgBDQDWQ</recordid><startdate>20241006</startdate><enddate>20241006</enddate><creator>Herrmann, Jordis Emilia</creator><creator>Gopinath, Aswath Mandakath</creator><creator>Norrlof, Mikael</creator><creator>Müller, Mark Niklas</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20241006</creationdate><title>Diagnosing Robotics Systems Issues with Large Language Models</title><author>Herrmann, Jordis Emilia ; Gopinath, Aswath Mandakath ; Norrlof, Mikael ; Müller, Mark Niklas</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-arxiv_primary_2410_090843</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Artificial Intelligence</topic><topic>Computer Science - Computation and Language</topic><topic>Computer Science - Learning</topic><topic>Computer Science - Robotics</topic><toplevel>online_resources</toplevel><creatorcontrib>Herrmann, Jordis Emilia</creatorcontrib><creatorcontrib>Gopinath, Aswath Mandakath</creatorcontrib><creatorcontrib>Norrlof, Mikael</creatorcontrib><creatorcontrib>Müller, Mark Niklas</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Herrmann, Jordis Emilia</au><au>Gopinath, Aswath Mandakath</au><au>Norrlof, Mikael</au><au>Müller, Mark Niklas</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Diagnosing Robotics Systems Issues with Large Language Models</atitle><date>2024-10-06</date><risdate>2024</risdate><abstract>Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.</abstract><doi>10.48550/arxiv.2410.09084</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2410.09084
ispartof
issn
language eng
recordid cdi_arxiv_primary_2410_09084
source arXiv.org
subjects Computer Science - Artificial Intelligence
Computer Science - Computation and Language
Computer Science - Learning
Computer Science - Robotics
title Diagnosing Robotics Systems Issues with Large Language Models
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T00%3A06%3A39IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Diagnosing%20Robotics%20Systems%20Issues%20with%20Large%20Language%20Models&rft.au=Herrmann,%20Jordis%20Emilia&rft.date=2024-10-06&rft_id=info:doi/10.48550/arxiv.2410.09084&rft_dat=%3Carxiv_GOX%3E2410_09084%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true