Evaluating Large Language Models for Automated CPT Code Prediction in Endovascular Neurosurgery

Large language models (LLMs) have been utilized to automate tasks like writing discharge summaries and operative reports in neurosurgery. The present study evaluates their ability to identify current procedural terminology (CPT) codes from operative reports. Three LLMs (ChatGPT 4.0, AtlasGPT and Gemini) were evaluated in their ability to provide CPT codes for diagnostic or interventional procedures in endovascular neurosurgery at a single institution. Responses were classified as correct, partially correct or incorrect, and the percentage of correctly identified CPT codes were calculated. The Chi-Square test and Kruskal Wallis test were used to compare responses across LLMs. A total of 30 operative notes were used in the present study. AtlasGPT identified CPT codes for 98.3% procedures with partially correct responses, while ChatGPT and Gemini provided partially correct responses for 86.7% and 30% procedures, respectively ( P