External validation of two clinical prediction models for mortality in COVID-19 patients (4C and NEWS2), in three centers in Medellín, Colombia: Assessing the impact of vaccination over time

•The mortality risk prediction tool 4C is more adequate, in comparison with NEWS2, as a predictor of in-hospital mortality in patients with COVID-19.•The 4C mortality risk score presented a good performance in clinically relevant metrics, with a negative likelihood ratio(LR) of 0.064 (CI 95% 0.01–0.26) for a score of 4 or less.•The operational characteristics of the prognostic model remained unchanged despite changes during the SARS-CoV2 pandemic, highlighted by vaccination and therapeutic interventions. External validation of the 4C and NEWS2 scores for the prediction of in-hospital mortality in COVID-19 patients, and evaluation of its operational performance in two time periods: before and after the start of the vaccination program in Colombia. Retrospective cohort in three high complexity hospitals in the city of Medellín, Colombia, between June 2020 and April 2022. The areas under the ROC curve (AUC) for the 4C mortality risk score and the NEWS2 were 0.75 (95% CI 0.73–0.78) and 0.68 (95% CI 0.66–0.71), respectively. For the 4C score, the AUC for the first and second periods was 0.77 (95% CI 0.74–0.80) and 0.75 (95% CI 0.71–0.78); whilst for the NEWS2 score, it was 0.68 (95% CI 0.65–0.71) and 0.69 (95% CI 0.64–0.73). The calibration for both scores was adequate, albeit with reduced performance during the second period. The 4C mortality risk score proved to be the more adequate predictor of in-hospital mortality in COVID-19 patients in this Latin American population. The operational performance during both time periods remained similar, which shows its utility notwithstanding major changes, including vaccination, as the pandemic evolved.