Checker Bug Detection and Repair in Deep Learning Libraries
Checker bugs in Deep Learning (DL) libraries are critical yet not well-explored. These bugs are often concealed in the input validation and error-checking code of DL libraries and can lead to silent failures, incorrect results, or unexpected program behavior in DL applications. Despite their potenti...
Gespeichert in:
Hauptverfasser: | , , , , , , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | |
container_volume | |
creator | Harzevili, Nima Shiri Mohajer, Mohammad Mahdi Shin, Jiho Wei, Moshi Uddin, Gias Yang, Jinqiu Wang, Junjie Wang, Song Ming, Zhen Jiang Nagappan, Nachiappan |
description | Checker bugs in Deep Learning (DL) libraries are critical yet not
well-explored. These bugs are often concealed in the input validation and
error-checking code of DL libraries and can lead to silent failures, incorrect
results, or unexpected program behavior in DL applications. Despite their
potential to significantly impact the reliability and performance of DL-enabled
systems built with these libraries, checker bugs have received limited
attention.
We present the first comprehensive study of DL checker bugs in two
widely-used DL libraries, i.e., TensorFlow and PyTorch. Initially, we
automatically collected a dataset of 2,418 commits from TensorFlow and PyTorch
repositories on GitHub from Sept. 2016 to Dec. 2023 using specific keywords
related to checker bugs. Through manual inspection, we identified 527 DL
checker bugs. Subsequently, we analyzed these bugs from three perspectives,
i.e., root causes, symptoms, and fixing patterns. Using the knowledge gained
via root cause analysis of checker bugs, we further propose TensorGuard, a
proof-of-concept RAG-based LLM-based tool to detect and fix checker bugs in DL
libraries via prompt engineering a series of ChatGPT prompts. We evaluated
TensorGuard's performance on a test dataset that includes 92 buggy and 135
clean checker-related changes in TensorFlow and PyTorch from January 2024 to
July 2024. Our results demonstrate that TensorGuard has high average recall
(94.51\%) using Chain of Thought prompting, a balanced performance between
precision and recall using Zero-Shot prompting and Few-Shot prompting
strategies. In terms of patch generation, TensorGuard achieves an accuracy of
11.1\%, which outperforms the state-of-the-art bug repair baseline by 2\%. We
have also applied TensorGuard on the latest six months' checker-related changes
(493 changes) of the JAX library from Google, which resulted in the detection
of 64 new checker bugs. |
doi_str_mv | 10.48550/arxiv.2410.06440 |
format | Article |
fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2410_06440</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2410_06440</sourcerecordid><originalsourceid>FETCH-arxiv_primary_2410_064403</originalsourceid><addsrcrecordid>eNpjYJA0NNAzsTA1NdBPLKrILNMzMgEKGJiZmBhwMlg7Z6QmZ6cWKTiVpiu4pJakJpdk5ucpJOalKASlFiRmFilk5gHFUwsUfFITi_Iy89IVfDKTihKLMlOLeRhY0xJzilN5oTQ3g7yba4izhy7YmviCoszcxKLKeJB18WDrjAmrAABPiDPY</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Checker Bug Detection and Repair in Deep Learning Libraries</title><source>arXiv.org</source><creator>Harzevili, Nima Shiri ; Mohajer, Mohammad Mahdi ; Shin, Jiho ; Wei, Moshi ; Uddin, Gias ; Yang, Jinqiu ; Wang, Junjie ; Wang, Song ; Ming, Zhen ; Jiang ; Nagappan, Nachiappan</creator><creatorcontrib>Harzevili, Nima Shiri ; Mohajer, Mohammad Mahdi ; Shin, Jiho ; Wei, Moshi ; Uddin, Gias ; Yang, Jinqiu ; Wang, Junjie ; Wang, Song ; Ming, Zhen ; Jiang ; Nagappan, Nachiappan</creatorcontrib><description>Checker bugs in Deep Learning (DL) libraries are critical yet not
well-explored. These bugs are often concealed in the input validation and
error-checking code of DL libraries and can lead to silent failures, incorrect
results, or unexpected program behavior in DL applications. Despite their
potential to significantly impact the reliability and performance of DL-enabled
systems built with these libraries, checker bugs have received limited
attention.
We present the first comprehensive study of DL checker bugs in two
widely-used DL libraries, i.e., TensorFlow and PyTorch. Initially, we
automatically collected a dataset of 2,418 commits from TensorFlow and PyTorch
repositories on GitHub from Sept. 2016 to Dec. 2023 using specific keywords
related to checker bugs. Through manual inspection, we identified 527 DL
checker bugs. Subsequently, we analyzed these bugs from three perspectives,
i.e., root causes, symptoms, and fixing patterns. Using the knowledge gained
via root cause analysis of checker bugs, we further propose TensorGuard, a
proof-of-concept RAG-based LLM-based tool to detect and fix checker bugs in DL
libraries via prompt engineering a series of ChatGPT prompts. We evaluated
TensorGuard's performance on a test dataset that includes 92 buggy and 135
clean checker-related changes in TensorFlow and PyTorch from January 2024 to
July 2024. Our results demonstrate that TensorGuard has high average recall
(94.51\%) using Chain of Thought prompting, a balanced performance between
precision and recall using Zero-Shot prompting and Few-Shot prompting
strategies. In terms of patch generation, TensorGuard achieves an accuracy of
11.1\%, which outperforms the state-of-the-art bug repair baseline by 2\%. We
have also applied TensorGuard on the latest six months' checker-related changes
(493 changes) of the JAX library from Google, which resulted in the detection
of 64 new checker bugs.</description><identifier>DOI: 10.48550/arxiv.2410.06440</identifier><language>eng</language><subject>Computer Science - Software Engineering</subject><creationdate>2024-10</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2410.06440$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2410.06440$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Harzevili, Nima Shiri</creatorcontrib><creatorcontrib>Mohajer, Mohammad Mahdi</creatorcontrib><creatorcontrib>Shin, Jiho</creatorcontrib><creatorcontrib>Wei, Moshi</creatorcontrib><creatorcontrib>Uddin, Gias</creatorcontrib><creatorcontrib>Yang, Jinqiu</creatorcontrib><creatorcontrib>Wang, Junjie</creatorcontrib><creatorcontrib>Wang, Song</creatorcontrib><creatorcontrib>Ming, Zhen</creatorcontrib><creatorcontrib>Jiang</creatorcontrib><creatorcontrib>Nagappan, Nachiappan</creatorcontrib><title>Checker Bug Detection and Repair in Deep Learning Libraries</title><description>Checker bugs in Deep Learning (DL) libraries are critical yet not
well-explored. These bugs are often concealed in the input validation and
error-checking code of DL libraries and can lead to silent failures, incorrect
results, or unexpected program behavior in DL applications. Despite their
potential to significantly impact the reliability and performance of DL-enabled
systems built with these libraries, checker bugs have received limited
attention.
We present the first comprehensive study of DL checker bugs in two
widely-used DL libraries, i.e., TensorFlow and PyTorch. Initially, we
automatically collected a dataset of 2,418 commits from TensorFlow and PyTorch
repositories on GitHub from Sept. 2016 to Dec. 2023 using specific keywords
related to checker bugs. Through manual inspection, we identified 527 DL
checker bugs. Subsequently, we analyzed these bugs from three perspectives,
i.e., root causes, symptoms, and fixing patterns. Using the knowledge gained
via root cause analysis of checker bugs, we further propose TensorGuard, a
proof-of-concept RAG-based LLM-based tool to detect and fix checker bugs in DL
libraries via prompt engineering a series of ChatGPT prompts. We evaluated
TensorGuard's performance on a test dataset that includes 92 buggy and 135
clean checker-related changes in TensorFlow and PyTorch from January 2024 to
July 2024. Our results demonstrate that TensorGuard has high average recall
(94.51\%) using Chain of Thought prompting, a balanced performance between
precision and recall using Zero-Shot prompting and Few-Shot prompting
strategies. In terms of patch generation, TensorGuard achieves an accuracy of
11.1\%, which outperforms the state-of-the-art bug repair baseline by 2\%. We
have also applied TensorGuard on the latest six months' checker-related changes
(493 changes) of the JAX library from Google, which resulted in the detection
of 64 new checker bugs.</description><subject>Computer Science - Software Engineering</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNpjYJA0NNAzsTA1NdBPLKrILNMzMgEKGJiZmBhwMlg7Z6QmZ6cWKTiVpiu4pJakJpdk5ucpJOalKASlFiRmFilk5gHFUwsUfFITi_Iy89IVfDKTihKLMlOLeRhY0xJzilN5oTQ3g7yba4izhy7YmviCoszcxKLKeJB18WDrjAmrAABPiDPY</recordid><startdate>20241008</startdate><enddate>20241008</enddate><creator>Harzevili, Nima Shiri</creator><creator>Mohajer, Mohammad Mahdi</creator><creator>Shin, Jiho</creator><creator>Wei, Moshi</creator><creator>Uddin, Gias</creator><creator>Yang, Jinqiu</creator><creator>Wang, Junjie</creator><creator>Wang, Song</creator><creator>Ming, Zhen</creator><creator>Jiang</creator><creator>Nagappan, Nachiappan</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20241008</creationdate><title>Checker Bug Detection and Repair in Deep Learning Libraries</title><author>Harzevili, Nima Shiri ; Mohajer, Mohammad Mahdi ; Shin, Jiho ; Wei, Moshi ; Uddin, Gias ; Yang, Jinqiu ; Wang, Junjie ; Wang, Song ; Ming, Zhen ; Jiang ; Nagappan, Nachiappan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-arxiv_primary_2410_064403</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Software Engineering</topic><toplevel>online_resources</toplevel><creatorcontrib>Harzevili, Nima Shiri</creatorcontrib><creatorcontrib>Mohajer, Mohammad Mahdi</creatorcontrib><creatorcontrib>Shin, Jiho</creatorcontrib><creatorcontrib>Wei, Moshi</creatorcontrib><creatorcontrib>Uddin, Gias</creatorcontrib><creatorcontrib>Yang, Jinqiu</creatorcontrib><creatorcontrib>Wang, Junjie</creatorcontrib><creatorcontrib>Wang, Song</creatorcontrib><creatorcontrib>Ming, Zhen</creatorcontrib><creatorcontrib>Jiang</creatorcontrib><creatorcontrib>Nagappan, Nachiappan</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Harzevili, Nima Shiri</au><au>Mohajer, Mohammad Mahdi</au><au>Shin, Jiho</au><au>Wei, Moshi</au><au>Uddin, Gias</au><au>Yang, Jinqiu</au><au>Wang, Junjie</au><au>Wang, Song</au><au>Ming, Zhen</au><au>Jiang</au><au>Nagappan, Nachiappan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Checker Bug Detection and Repair in Deep Learning Libraries</atitle><date>2024-10-08</date><risdate>2024</risdate><abstract>Checker bugs in Deep Learning (DL) libraries are critical yet not
well-explored. These bugs are often concealed in the input validation and
error-checking code of DL libraries and can lead to silent failures, incorrect
results, or unexpected program behavior in DL applications. Despite their
potential to significantly impact the reliability and performance of DL-enabled
systems built with these libraries, checker bugs have received limited
attention.
We present the first comprehensive study of DL checker bugs in two
widely-used DL libraries, i.e., TensorFlow and PyTorch. Initially, we
automatically collected a dataset of 2,418 commits from TensorFlow and PyTorch
repositories on GitHub from Sept. 2016 to Dec. 2023 using specific keywords
related to checker bugs. Through manual inspection, we identified 527 DL
checker bugs. Subsequently, we analyzed these bugs from three perspectives,
i.e., root causes, symptoms, and fixing patterns. Using the knowledge gained
via root cause analysis of checker bugs, we further propose TensorGuard, a
proof-of-concept RAG-based LLM-based tool to detect and fix checker bugs in DL
libraries via prompt engineering a series of ChatGPT prompts. We evaluated
TensorGuard's performance on a test dataset that includes 92 buggy and 135
clean checker-related changes in TensorFlow and PyTorch from January 2024 to
July 2024. Our results demonstrate that TensorGuard has high average recall
(94.51\%) using Chain of Thought prompting, a balanced performance between
precision and recall using Zero-Shot prompting and Few-Shot prompting
strategies. In terms of patch generation, TensorGuard achieves an accuracy of
11.1\%, which outperforms the state-of-the-art bug repair baseline by 2\%. We
have also applied TensorGuard on the latest six months' checker-related changes
(493 changes) of the JAX library from Google, which resulted in the detection
of 64 new checker bugs.</abstract><doi>10.48550/arxiv.2410.06440</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext_linktorsrc |
identifier | DOI: 10.48550/arxiv.2410.06440 |
ispartof | |
issn | |
language | eng |
recordid | cdi_arxiv_primary_2410_06440 |
source | arXiv.org |
subjects | Computer Science - Software Engineering |
title | Checker Bug Detection and Repair in Deep Learning Libraries |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T08%3A34%3A01IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Checker%20Bug%20Detection%20and%20Repair%20in%20Deep%20Learning%20Libraries&rft.au=Harzevili,%20Nima%20Shiri&rft.date=2024-10-08&rft_id=info:doi/10.48550/arxiv.2410.06440&rft_dat=%3Carxiv_GOX%3E2410_06440%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |