Does Ignoring Model Selection When Assessing the Effect of Particulate Matter Air Pollution on Mortality Make Us Too Vigilant?

Purpose To investigate the extent to which standard errors can be underestimated in time-series studies of the association between particulate matter air pollution (PM) and mortality if model selection variation is not accounted for. Methods Actual-time series data from Cook County, Illinois, and Salt Lake County, Utah, for the period 1987 to 2000 were used to generate mortality time series. These series were used to examine the overconfidence resulting from ignoring variability introduced by the model selection process. Results When variation associated with a model selection process is not accounted for, we found that the estimated standard errors for the effect of PM on mortality were substantially smaller than the true standard errors that necessarily incorporate model selection variability. Because of this, the true standard errors are approximately 70% larger than the reported standard errors. We also found that not accounting for model selection effects can result in the observed size of tests of no association between PM and mortality being up to about five times the nominal significance level. Conclusions Failing to account properly for the effect of model selection can reduce the accepted burden of proof for concluding a statistically significant association between PM and mortality.