Evaluating Physician-AI Interaction for Multiple Myeloma Management: Paving the Path Towards Precision Oncology

Introduction The gap between clinical trials and real-world care continues to grow in the field of multiple myeloma (MM). Randomized controlled trials (RCTs) cannot account for every patient scenario. Modern clinical decision support systems (CDSSs) that leverage machine learning (ML) models offer an alternate path to precision oncology. However, little work has been done to understand how clinicians reconcile RCT with ML data, particularly when results are discordant. Aim We designed a CDSS that displays simulated survival and adverse event data from an RCT and ML model and conducted a survey study to evaluate how clinicians incorporate the available data to make treatment decisions for 12 patients with MM. Methods Participants were presented with varying combinations of RCT and ML results in increasing “tiers” of information for 12 patients (A-L) with MM (Table 1). In tier 1, participants were provided RCT data only. In tier 2, participants were given outcomes of an ML model and in tier 3, they were provided with information about how the model was trained and validated. At each tier, participants were asked to select a treatment (“red pill” or “blue pill”), rate their confidence in treatment on a Likert scale from 1-10, and when ML data was available, rate their perceived reliability of the model on a Likert scale from 1-10. Participants were recruited from internal medicine and hematology/oncology departments via email between January and April 2023, and were offered a $50 Amazon gift card as incentive. We used descriptive statistics to analyze respondent characteristics. For each scenario, we ran two-sample paired t-tests to compare the change in confidence and reliability between tier 2 and tier 1 (ML versus RCT data) and tier 3 versus tier 2 (ML data with information about training and validation versus without). We utilized a Bonferroni correction to adjust the alpha level for significance. We also ran McNemar's tests with a Bonferroni correction to assess the difference in proportions of blue pill selection at different tiers to characterize the extent of treatment switching. Results A total of 284 physicians were invited to participate in the study and 32 participated, for a response rate of 11.3%. Half were internal medicine residents and half were hematology/oncology fellows and attendings. A majority were male (72.0%), white (69.0%) and all were less than 40 years of age. For scenarios A-D, the patient met inclusion criteria for the RCT and wa