‘Data snooping’ and the significance level in multiple testing

Carrying out a number of statistical tests, using some or all of the same data repeatedly, increases the chance of making decision errors. If, moreover, the tests to be done are formulated after looking at the results of an interim analysis using the same data – a situation termed ‘data snooping’ – there is serious risk of biasing the entire investigation. Data snooping can be done in a controlled and reasonable way, or in an uncontrolled and misleading (and even unethical) way. Several consequences of, and potential solutions to, the problem of data snooping are explained.