Predicting alcohol dependence treatment outcomes: a prospective comparative study of clinical psychologists versus ‘trained’ machine learning models

Background and aims Clinical staff are typically poor at predicting alcohol dependence treatment outcomes. Machine learning (ML) offers the potential to model complex clinical data more effectively. This study tested the predictive accuracy of ML algorithms demonstrated to be effective in predicting alcohol dependence outcomes, compared with clinical judgement and traditional linear regression. Design Prospective study. ML models were trained on 1016 previously treated patients (training‐set) who attended a hospital‐based alcohol and drug clinic. ML models (n = 27), clinical psychologists (n = 10) and a ‘traditional’ logistic regression model (n = 1) predicted treatment outcome during the initial treatment session of an alcohol dependence programme. Setting A 12‐week cognitive behavioural therapy (CBT)‐based abstinence programme for alcohol dependence in a hospital‐based alcohol and drug clinic in Australia. Participants Prospective predictions were made for 220 new patients (test‐set; 70.91% male, mean age = 35.78 years, standard deviation = 9.19). Sixty‐nine (31.36%) patients successfully completed treatment. Measurements Treatment success was the primary outcome variable. The cross‐validated training‐set accuracy of ML models was used to determine optimal parameters for selecting models for prospective prediction. Accuracy, sensitivity, specificity, area under the receiver operator curve (AUC), Brier score and calibration curves were calculated and compared across predictions. Findings The mean aggregate accuracy of the ML models (63.06%) was higher than the mean accuracy of psychologist predictions (56.36%). The most accurate ML model achieved 70% accuracy, as did logistic regression. Both were more accurate than psychologists (P