Classifying injury narratives of large administrative databases for surveillance—A practical approach combining machine learning ensembles and human review

•Manual classification of the cause/events leading to injury is useful for injury prevention but can be prohibitive for large batches of narratives.•Human-machine learning ensemble approaches maximize the accuracy of the machine-assigned codes allowing strategic filtering for manual review.•If resources are constrained at a low level, the best approach for accuracy is to combine manual coding with codes assigned by the LR algorithm alone.•An ensemble approach to filtering affords more confidence in classifications if models make predictions in fundamentally different ways.•Coding rarer events accurately requires sophisticated filtering or integration of highly tailored resource- intensive methods such as NLP. Injury narratives are now available real time and include useful information for injury surveillance and prevention. However, manual classification of the cause or events leading to injury found in large batches of narratives, such as workers compensation claims databases, can be prohibitive. In this study we compare the utility of four machine learning algorithms (Naïve Bayes, Single word and Bi-gram models, Support Vector Machine and Logistic Regression) for classifying narratives into Bureau of Labor Statistics Occupational Injury and Illness event leading to injury classifications for a large workers compensation database. These algorithms are known to do well classifying narrative text and are fairly easy to implement with off-the-shelf software packages such as Python. We propose human-machine learning ensemble approaches which maximize the power and accuracy of the algorithms for machine-assigned codes and allow for strategic filtering of rare, emerging or ambiguous narratives for manual review. We compare human-machine approaches based on filtering on the prediction strength of the classifier vs. agreement between algorithms. Regularized Logistic Regression (LR) was the best performing algorithm alone. Using this algorithm and filtering out the bottom 30% of predictions for manual review resulted in high accuracy (overall sensitivity/positive predictive value of 0.89) of the final machine-human coded dataset. The best pairings of algorithms included Naïve Bayes with Support Vector Machine whereby the triple ensemble NBSW=NBBI-GRAM=SVM had very high performance (0.93 overall sensitivity/positive predictive value and high accuracy (i.e. high sensitivity and positive predictive values)) across both large and small categories leaving 41% of the na