An Empirical Study on Machine Learning Model Calibration for Chest Radiograph Triage

Despite ongoing efforts on improving machine learning model performance, capturing model uncertainty remains a major challenge in medical imaging, an important subject that has been largely overlooked by the community. This work borrows standard model calibration approaches and empirically demonstrates their effectiveness on medical imaging triage with labels automatically extracted by different natural language processing techniques. We demonstrate both the strength and weakness of three different calibration methods using two sets of NLP labels. The tests are conducted on human-labelled ground truth. Although all methods yield comparable results, our proposed approach further improves AUCs when paired with a strong NLP model that generates smooth labels.