Improving the transparency of statistical reporting in Conservation Letters

Conversely, undisclosed analysis practices such as cherry picking “significant” results and p‐hacking (e.g., making decisions about sampling stopping rules, treatment of outliers, transformations, and/or analysis techniques based on whether results meet or fail to meet a statistical significance threshold) have been directly linked to the inability to replicate many important, published experimental effects (Fidler et al., ; Forstmeier, Wagenmakers, & Parker, ; Simmons, Nelson, & Simonsohn, ). If you are also reporting the outcomes of null hypothesis significance tests (i.e., p values), below are some further important messages. 4 MESSAGE NO. 4: STATE THE SAMPLING STOPPING RULE ASSOCIATED WITH YOUR HYPOTHESIS TEST Often, there are practical constraints on sample size, and therefore statistical power. A study result is compelling evidence of an effect only if the effect is large enough to be ecologically or theoretically interesting and unusual enough not to have arisen by chance. 7 MESSAGE NO. 7: LOOK OUT FOR LESS OBVIOUS INSTANCES OF NULL HYPOTHESIS TESTING; MESSAGE NOs. 4 TO 6 APPLY TO THEM TOO The messages above are not only relevant to researchers conducting t‐tests and ANOVAs as their core analyses. On closer inspection, many such cases do involve null hypothesis testing as part of a larger procedure, for example, parameters selected for inclusion in models on the grounds that they reached p < .05, or goodness‐of‐fit statistics later subjected to statistical significance analysis.