Predicting 30-day risk from benzodiazepine/Z-drug dispensations in older adults using administrative data: A prognostic machine learning approach

•This study considered the perspective of health regulators and their mandate to monitor adverse outcomes related to benzodiazepine use in older adults.•Machine learning (ML) prediction methods can assist prescription drug monitoring programs.•Our analysis included informative clinical metrics and simulations to assess ML prediction performance which included measures to describe ML explainability related to our prediction model.•Although our data included complete capture of admissions, physician claims and prescription drug histories, we were unable to include measurable social factors data.•Our findings show that health regulators would require additional types of data if they are to incorporate ML approaches into their drug monitoring programs as our ML predictions were generally uninformative; predicting adverse outcomes in older adults is generally a difficult undertaking. To develop a machine-learning (ML) model using administrative data to estimate risk of adverse outcomes within 30-days of a benzodiazepine (BZRA) dispensation in older adults for use by health departments/regulators. This study was conducted in Alberta, Canada during 2018–2019 in Albertans 65 years of age and older. Those with any history of malignancy or palliative care were excluded. Each BZRA dispensation from a community pharmacy served as the unit of analysis. ML algorithms were developed on 2018 administrative data to predict risk of any-cause hospitalization, emergency department visit or death within 30-days of a BZRA dispensation. Validation on 2019 administrative data was done using XGBoost to evaluate discrimination, calibration and other relevant metrics on ranked predictions. Daily and quarterly predictions were simulated on 2019 data. 65,063 study participants were included which represented 633,333 BZRA dispensation during 2018–2019. The validation set had 314,615 dispensations linked to 55,928 all-cause outcomes representing a pre-test probability of 17.8%. C-statistic for the XGBoost model was 0.75. Measuring risk at the end of 2019, the top 0.1 percentile of predicted risk had a LR + of 40.31 translating to a post-test probability of 90%. Daily and quarterly classification simulations resulted in uninformative predictions with positive likelihood ratios less than 10 in all risk prediction categories. Previous history of admissions was ranked highest in variable importance. Developing ML models using only administrative health data may not provide health regulators