Retrospective Evaluation of Artificial Intelligence Leveraging Free-Text Imaging Order Entry to Facilitate Federally Required Clinical Decision Support

The Protecting Access to Medicare Act mandates clinical decision support (CDS) at imaging order entry, necessitating the use of structured indications to map CDS scores. We evaluated the performance of a commercially available artificial intelligence (AI) tool leveraging free-text order entry to facilitate provider selection of the necessary structured indications. Our institution implemented an AI tool offering predicted structured indications based upon the ordering provider’s entry of a free-text reason for examination. Providers remained able to order via the traditional direct search for structured indications. Alternatively, they could take the new free-text–AI approach allowing them to select from AI-predicted indications, perform additional direct searches, indicate no matching indication, or exit CDS workflow. We hypothesized the free-text–AI approach would be elected more often and the AI tool would be successful in facilitating selection of structured indications. We reviewed advanced imaging orders (n = 40,053) for the first 3 months (February to May 2020) since implementation. Providers were more likely (P < .001) to choose the free-text–AI approach (23,580; 58.9%) to order entry over direct search for structured indications (16,473; 41.1%). The AI tool yielded alerts with predicted indications in 91.7% (n = 21,631) of orders with free text. Ultimately, providers chose AI-predicted indications in 57.7% (n = 12,490) of cases in which they were offered by the tool. Providers significantly more often elected the new free-text–AI approach to order entry for CDS, suggesting provider preference over the traditional approach. The AI tool commonly predicted indications acceptable to ordering providers.