Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery
Large language models (LLMs) have been utilized to automate tasks like writing discharge summaries and operative reports in neurosurgery. The present study evaluates their ability to identify current procedural terminology (CPT) codes from operative reports. Three LLMs (ChatGPT 4.0, AtlasGPT and Gem...
Gespeichert in:
| Veröffentlicht in: | Journal of medical systems 2025-01, Vol.49 (1), p.15, Article 15 |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | 1 |
| container_start_page | 15 |
| container_title | Journal of medical systems |
| container_volume | 49 |
| creator | Roy, Joanna M. Self, D. Mitchell Isch, Emily Musmar, Basel Lan, Matthews Keppetipola, Kavantissa Koduri, Sravanthi Pontarelli, Mary-Katharine Tjoumakaris, Stavropoula I. Gooch, M. Reid Rosenwasser, Robert H. Jabbour, Pascal M. |
| description | Large language models (LLMs) have been utilized to automate tasks like writing discharge summaries and operative reports in neurosurgery. The present study evaluates their ability to identify current procedural terminology (CPT) codes from operative reports. Three LLMs (ChatGPT 4.0, AtlasGPT and Gemini) were evaluated in their ability to provide CPT codes for diagnostic or interventional procedures in endovascular neurosurgery at a single institution. Responses were classified as correct, partially correct or incorrect, and the percentage of correctly identified CPT codes were calculated. The Chi-Square test and Kruskal Wallis test were used to compare responses across LLMs. A total of 30 operative notes were used in the present study. AtlasGPT identified CPT codes for 98.3% procedures with partially correct responses, while ChatGPT and Gemini provided partially correct responses for 86.7% and 30% procedures, respectively (
P
|
| doi_str_mv | 10.1007/s10916-025-02149-4 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_miscellaneous_3159688992</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3159688992</sourcerecordid><originalsourceid>FETCH-LOGICAL-c256t-69cf482f9ff51de66e88caabd25194550df5c307004f66e645a54b3edf59ad263</originalsourceid><addsrcrecordid>eNp9kM1PwyAYxonRuDn9BzwYEi9eqtAWWo5LMz-SqTvMxBthBZYubZlQluy_l63zIx48AG94fu_zwgPAJUa3GKHszmHEMI1QTMLCKYvSIzDEJEsimrP341_1AJw5t0IIMUqzUzBIWE4SisgQ8MlG1F50VbuEU2GXKuzt0otQPBupage1sXDsO9OITklYzOawCAKcWSWrsqtMC6sWTlppNsKVvhYWvihvjfPBzG7PwYkWtVMXh3ME3u4n8-Ixmr4-PBXjaVTGhHYRZaVO81gzrQmWilKV56UQCxkTzFJCkNSkTFCGUKqDSFMiSLpIVLhmQsY0GYGb3ndtzYdXruNN5UpV16JVxjueYMJonjMWB_T6D7oy3rbhdXuK5DGiOyruqTL8xVml-dpWjbBbjhHfxc_7-HmIn-_j52loujpY-0Wj5HfLV94BSHrABakNAf3M_sf2EztvkB0</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3159582062</pqid></control><display><type>article</type><title>Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery</title><source>MEDLINE</source><source>SpringerLink Journals</source><creator>Roy, Joanna M. ; Self, D. Mitchell ; Isch, Emily ; Musmar, Basel ; Lan, Matthews ; Keppetipola, Kavantissa ; Koduri, Sravanthi ; Pontarelli, Mary-Katharine ; Tjoumakaris, Stavropoula I. ; Gooch, M. Reid ; Rosenwasser, Robert H. ; Jabbour, Pascal M.</creator><creatorcontrib>Roy, Joanna M. ; Self, D. Mitchell ; Isch, Emily ; Musmar, Basel ; Lan, Matthews ; Keppetipola, Kavantissa ; Koduri, Sravanthi ; Pontarelli, Mary-Katharine ; Tjoumakaris, Stavropoula I. ; Gooch, M. Reid ; Rosenwasser, Robert H. ; Jabbour, Pascal M.</creatorcontrib><description>Large language models (LLMs) have been utilized to automate tasks like writing discharge summaries and operative reports in neurosurgery. The present study evaluates their ability to identify current procedural terminology (CPT) codes from operative reports. Three LLMs (ChatGPT 4.0, AtlasGPT and Gemini) were evaluated in their ability to provide CPT codes for diagnostic or interventional procedures in endovascular neurosurgery at a single institution. Responses were classified as correct, partially correct or incorrect, and the percentage of correctly identified CPT codes were calculated. The Chi-Square test and Kruskal Wallis test were used to compare responses across LLMs. A total of 30 operative notes were used in the present study. AtlasGPT identified CPT codes for 98.3% procedures with partially correct responses, while ChatGPT and Gemini provided partially correct responses for 86.7% and 30% procedures, respectively (
P
< 0.001). AtlasGPT identified CPT codes correctly in an average of 35.3% of procedures, followed by ChatGPT (35.1%) and Gemini (8.9%) (
P
< 0.001). A pairwise comparison among three LLMs revealed that AtlasGPT and ChatGPT outperformed Gemini. Untrained LLMs have the ability to identify partially correct CPT codes in endovascular neurosurgery. Training these models could further enhance their ability to identify CPT codes and minimize healthcare expenditure.</description><identifier>ISSN: 1573-689X</identifier><identifier>ISSN: 0148-5598</identifier><identifier>EISSN: 1573-689X</identifier><identifier>DOI: 10.1007/s10916-025-02149-4</identifier><identifier>PMID: 39853605</identifier><language>eng</language><publisher>New York: Springer US</publisher><subject>Artificial intelligence ; Automation ; Cardiovascular system ; Chatbots ; Chi-square test ; Codes ; Current Procedural Terminology ; Endovascular Procedures - methods ; Health care expenditures ; Health Informatics ; Health Sciences ; Humans ; Intervention ; Large language models ; Machine learning ; Medical coding ; Medical diagnosis ; Medicine ; Medicine & Public Health ; Neurosurgery ; Neurosurgical Procedures - methods ; Statistics for Life Sciences</subject><ispartof>Journal of medical systems, 2025-01, Vol.49 (1), p.15, Article 15</ispartof><rights>The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2025 Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</rights><rights>2025. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.</rights><rights>Copyright Springer Nature B.V. Dec 2025</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c256t-69cf482f9ff51de66e88caabd25194550df5c307004f66e645a54b3edf59ad263</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://link.springer.com/content/pdf/10.1007/s10916-025-02149-4$$EPDF$$P50$$Gspringer$$H</linktopdf><linktohtml>$$Uhttps://link.springer.com/10.1007/s10916-025-02149-4$$EHTML$$P50$$Gspringer$$H</linktohtml><link.rule.ids>314,776,780,27901,27902,41464,42533,51294</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/39853605$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Roy, Joanna M.</creatorcontrib><creatorcontrib>Self, D. Mitchell</creatorcontrib><creatorcontrib>Isch, Emily</creatorcontrib><creatorcontrib>Musmar, Basel</creatorcontrib><creatorcontrib>Lan, Matthews</creatorcontrib><creatorcontrib>Keppetipola, Kavantissa</creatorcontrib><creatorcontrib>Koduri, Sravanthi</creatorcontrib><creatorcontrib>Pontarelli, Mary-Katharine</creatorcontrib><creatorcontrib>Tjoumakaris, Stavropoula I.</creatorcontrib><creatorcontrib>Gooch, M. Reid</creatorcontrib><creatorcontrib>Rosenwasser, Robert H.</creatorcontrib><creatorcontrib>Jabbour, Pascal M.</creatorcontrib><title>Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery</title><title>Journal of medical systems</title><addtitle>J Med Syst</addtitle><addtitle>J Med Syst</addtitle><description>Large language models (LLMs) have been utilized to automate tasks like writing discharge summaries and operative reports in neurosurgery. The present study evaluates their ability to identify current procedural terminology (CPT) codes from operative reports. Three LLMs (ChatGPT 4.0, AtlasGPT and Gemini) were evaluated in their ability to provide CPT codes for diagnostic or interventional procedures in endovascular neurosurgery at a single institution. Responses were classified as correct, partially correct or incorrect, and the percentage of correctly identified CPT codes were calculated. The Chi-Square test and Kruskal Wallis test were used to compare responses across LLMs. A total of 30 operative notes were used in the present study. AtlasGPT identified CPT codes for 98.3% procedures with partially correct responses, while ChatGPT and Gemini provided partially correct responses for 86.7% and 30% procedures, respectively (
P
< 0.001). AtlasGPT identified CPT codes correctly in an average of 35.3% of procedures, followed by ChatGPT (35.1%) and Gemini (8.9%) (
P
< 0.001). A pairwise comparison among three LLMs revealed that AtlasGPT and ChatGPT outperformed Gemini. Untrained LLMs have the ability to identify partially correct CPT codes in endovascular neurosurgery. Training these models could further enhance their ability to identify CPT codes and minimize healthcare expenditure.</description><subject>Artificial intelligence</subject><subject>Automation</subject><subject>Cardiovascular system</subject><subject>Chatbots</subject><subject>Chi-square test</subject><subject>Codes</subject><subject>Current Procedural Terminology</subject><subject>Endovascular Procedures - methods</subject><subject>Health care expenditures</subject><subject>Health Informatics</subject><subject>Health Sciences</subject><subject>Humans</subject><subject>Intervention</subject><subject>Large language models</subject><subject>Machine learning</subject><subject>Medical coding</subject><subject>Medical diagnosis</subject><subject>Medicine</subject><subject>Medicine & Public Health</subject><subject>Neurosurgery</subject><subject>Neurosurgical Procedures - methods</subject><subject>Statistics for Life Sciences</subject><issn>1573-689X</issn><issn>0148-5598</issn><issn>1573-689X</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2025</creationdate><recordtype>article</recordtype><sourceid>EIF</sourceid><recordid>eNp9kM1PwyAYxonRuDn9BzwYEi9eqtAWWo5LMz-SqTvMxBthBZYubZlQluy_l63zIx48AG94fu_zwgPAJUa3GKHszmHEMI1QTMLCKYvSIzDEJEsimrP341_1AJw5t0IIMUqzUzBIWE4SisgQ8MlG1F50VbuEU2GXKuzt0otQPBupage1sXDsO9OITklYzOawCAKcWSWrsqtMC6sWTlppNsKVvhYWvihvjfPBzG7PwYkWtVMXh3ME3u4n8-Ixmr4-PBXjaVTGhHYRZaVO81gzrQmWilKV56UQCxkTzFJCkNSkTFCGUKqDSFMiSLpIVLhmQsY0GYGb3ndtzYdXruNN5UpV16JVxjueYMJonjMWB_T6D7oy3rbhdXuK5DGiOyruqTL8xVml-dpWjbBbjhHfxc_7-HmIn-_j52loujpY-0Wj5HfLV94BSHrABakNAf3M_sf2EztvkB0</recordid><startdate>20250124</startdate><enddate>20250124</enddate><creator>Roy, Joanna M.</creator><creator>Self, D. Mitchell</creator><creator>Isch, Emily</creator><creator>Musmar, Basel</creator><creator>Lan, Matthews</creator><creator>Keppetipola, Kavantissa</creator><creator>Koduri, Sravanthi</creator><creator>Pontarelli, Mary-Katharine</creator><creator>Tjoumakaris, Stavropoula I.</creator><creator>Gooch, M. Reid</creator><creator>Rosenwasser, Robert H.</creator><creator>Jabbour, Pascal M.</creator><general>Springer US</general><general>Springer Nature B.V</general><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QQ</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TK</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>H8G</scope><scope>JG9</scope><scope>JQ2</scope><scope>K9.</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>NAPCQ</scope><scope>P64</scope><scope>7X8</scope></search><sort><creationdate>20250124</creationdate><title>Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery</title><author>Roy, Joanna M. ; Self, D. Mitchell ; Isch, Emily ; Musmar, Basel ; Lan, Matthews ; Keppetipola, Kavantissa ; Koduri, Sravanthi ; Pontarelli, Mary-Katharine ; Tjoumakaris, Stavropoula I. ; Gooch, M. Reid ; Rosenwasser, Robert H. ; Jabbour, Pascal M.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c256t-69cf482f9ff51de66e88caabd25194550df5c307004f66e645a54b3edf59ad263</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2025</creationdate><topic>Artificial intelligence</topic><topic>Automation</topic><topic>Cardiovascular system</topic><topic>Chatbots</topic><topic>Chi-square test</topic><topic>Codes</topic><topic>Current Procedural Terminology</topic><topic>Endovascular Procedures - methods</topic><topic>Health care expenditures</topic><topic>Health Informatics</topic><topic>Health Sciences</topic><topic>Humans</topic><topic>Intervention</topic><topic>Large language models</topic><topic>Machine learning</topic><topic>Medical coding</topic><topic>Medical diagnosis</topic><topic>Medicine</topic><topic>Medicine & Public Health</topic><topic>Neurosurgery</topic><topic>Neurosurgical Procedures - methods</topic><topic>Statistics for Life Sciences</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Roy, Joanna M.</creatorcontrib><creatorcontrib>Self, D. Mitchell</creatorcontrib><creatorcontrib>Isch, Emily</creatorcontrib><creatorcontrib>Musmar, Basel</creatorcontrib><creatorcontrib>Lan, Matthews</creatorcontrib><creatorcontrib>Keppetipola, Kavantissa</creatorcontrib><creatorcontrib>Koduri, Sravanthi</creatorcontrib><creatorcontrib>Pontarelli, Mary-Katharine</creatorcontrib><creatorcontrib>Tjoumakaris, Stavropoula I.</creatorcontrib><creatorcontrib>Gooch, M. Reid</creatorcontrib><creatorcontrib>Rosenwasser, Robert H.</creatorcontrib><creatorcontrib>Jabbour, Pascal M.</creatorcontrib><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical & Transportation Engineering Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology & Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Copper Technical Reference Library</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>ProQuest Health & Medical Complete (Alumni)</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Nursing & Allied Health Premium</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><jtitle>Journal of medical systems</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Roy, Joanna M.</au><au>Self, D. Mitchell</au><au>Isch, Emily</au><au>Musmar, Basel</au><au>Lan, Matthews</au><au>Keppetipola, Kavantissa</au><au>Koduri, Sravanthi</au><au>Pontarelli, Mary-Katharine</au><au>Tjoumakaris, Stavropoula I.</au><au>Gooch, M. Reid</au><au>Rosenwasser, Robert H.</au><au>Jabbour, Pascal M.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery</atitle><jtitle>Journal of medical systems</jtitle><stitle>J Med Syst</stitle><addtitle>J Med Syst</addtitle><date>2025-01-24</date><risdate>2025</risdate><volume>49</volume><issue>1</issue><spage>15</spage><pages>15-</pages><artnum>15</artnum><issn>1573-689X</issn><issn>0148-5598</issn><eissn>1573-689X</eissn><abstract>Large language models (LLMs) have been utilized to automate tasks like writing discharge summaries and operative reports in neurosurgery. The present study evaluates their ability to identify current procedural terminology (CPT) codes from operative reports. Three LLMs (ChatGPT 4.0, AtlasGPT and Gemini) were evaluated in their ability to provide CPT codes for diagnostic or interventional procedures in endovascular neurosurgery at a single institution. Responses were classified as correct, partially correct or incorrect, and the percentage of correctly identified CPT codes were calculated. The Chi-Square test and Kruskal Wallis test were used to compare responses across LLMs. A total of 30 operative notes were used in the present study. AtlasGPT identified CPT codes for 98.3% procedures with partially correct responses, while ChatGPT and Gemini provided partially correct responses for 86.7% and 30% procedures, respectively (
P
< 0.001). AtlasGPT identified CPT codes correctly in an average of 35.3% of procedures, followed by ChatGPT (35.1%) and Gemini (8.9%) (
P
< 0.001). A pairwise comparison among three LLMs revealed that AtlasGPT and ChatGPT outperformed Gemini. Untrained LLMs have the ability to identify partially correct CPT codes in endovascular neurosurgery. Training these models could further enhance their ability to identify CPT codes and minimize healthcare expenditure.</abstract><cop>New York</cop><pub>Springer US</pub><pmid>39853605</pmid><doi>10.1007/s10916-025-02149-4</doi></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1573-689X |
| ispartof | Journal of medical systems, 2025-01, Vol.49 (1), p.15, Article 15 |
| issn | 1573-689X 0148-5598 1573-689X |
| language | eng |
| recordid | cdi_proquest_miscellaneous_3159688992 |
| source | MEDLINE; SpringerLink Journals |
| subjects | Artificial intelligence Automation Cardiovascular system Chatbots Chi-square test Codes Current Procedural Terminology Endovascular Procedures - methods Health care expenditures Health Informatics Health Sciences Humans Intervention Large language models Machine learning Medical coding Medical diagnosis Medicine Medicine & Public Health Neurosurgery Neurosurgical Procedures - methods Statistics for Life Sciences |
| title | Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-15T08%3A11%3A10IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Evaluating%20Large%20Language%20Models%20for%20Automated%20CPT%20Code%20Prediction%20in%20Endovascular%20Neurosurgery&rft.jtitle=Journal%20of%20medical%20systems&rft.au=Roy,%20Joanna%20M.&rft.date=2025-01-24&rft.volume=49&rft.issue=1&rft.spage=15&rft.pages=15-&rft.artnum=15&rft.issn=1573-689X&rft.eissn=1573-689X&rft_id=info:doi/10.1007/s10916-025-02149-4&rft_dat=%3Cproquest_cross%3E3159688992%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3159582062&rft_id=info:pmid/39853605&rfr_iscdi=true |