Spurious early ecological association suggesting BCG vaccination effectiveness for COVID-19

Several ecologic studies have suggested that the bacillus Calmette-Guérin (BCG) vaccine may be protective against SARS-CoV-2 infection including a highly-cited published pre-print by Miller et al., finding that middle/high- and high-income countries that never had a universal BCG policy experienced higher COVID-19 burden compared to countries that currently have universal BCG vaccination policies. We provide a case study of the limitations of ecologic analyses by evaluating whether these early ecologic findings persisted as the pandemic progressed. Similar to Miller et al., we employed Wilcoxon Rank Sum Tests to compare population medians in COVID-19 mortality, incidence, and mortality-to-incidence ratio between countries with universal BCG policies compared to those that never had such policies. We then computed Pearson's r correlations to evaluate the association between year of BCG vaccination policy implementation and COVID-19 outcomes. We repeated these analyses for every month in 2020 subsequent to Miller et al.'s March 2020 analysis. We found that the differences in COVID-19 burden associated with BCG vaccination policies in March 2020 generally diminished in magnitude and usually lost statistical significance as the pandemic progressed. While six of nine analyses were statistically significant in March, only two were significant by the end of 2020. These results underscore the need for caution in interpreting ecologic studies, given their inherent methodological limitations, which can be magnified in the context of a rapidly evolving pandemic in which there is measurement error of both exposure and outcome status.