Statistical Diversions

The term "data snooping" refers to the practice of choosing which statistical analyses to apply to a set of data after having first looked at those data. Data snooping contradicts a fundamental precept of applied statistics, that the scheme of analysis is to be planned in advance. In this column, the authors shall elucidate the statistical consequences of data snooping in the context of hypothesis testing. They shall show why statisticians should be wary of this practice and, yet, why it is almost impossible to avoid. Answers to their previous five questions and five new questions related to data snooping are also presented.