P < .05 is in the Eye of the Beholder: A Response to Beaujean and Farmer (2020)

The recent commentary by Beaujean and Farmer (2020) on the original paper by Dixon et al. (2019) serves a cautionary tale of selective p-values, the law of small N sizes, and the type-II error. We believe these authors have crafted a somewhat questionable argument in which only 57% of the original Dixon et al. data were re-analyzed, based on a series of assumptions that removed significant power from the original statistical analysis. We here provide an additional 12 re-analyses of the original data and demonstrated that depending on the testing assumptions and data inclusion criteria, p-value may or may not exceed the commonly used .05 level. Although we support healthy discussion on experimental methods, and appreciate the reinterpretation of our findings, the reader is cautioned that because Beaujean and Farmer’s conclusions are limited to such a restricted range of data and different pre-analytic assumptions, the actual importance of their obtained p -value must remain in the eye of the beholder.