Virtual reality check: Statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments

Virtual reality (VR) is a popular subject of scientific study across a variety of academic fields. In the present study we evaluate methodological trends in behavioral research on VR with respect to data collection practices, statistical reporting, and data availability. In line with this goal, we conducted a meta-scientific analysis of 61 articles encompassing a total of 1122 statistical tests and highlight three emergent trends that inform our understanding of past and future studies focused on VR. Conclusions from analysis of the data include a high incidence of errors in statistical reporting, and a general lack of transparency with respect to the availability of study data. Transparency in data analysis, increased statistical power, and more careful reporting of statistical outcomes are suggested to heighten methodological rigor and improve reproducibility in the field of VR research. •We conducted a systematic review of empirical research on VR.•We included 61 articles reporting 1122 statistical tests.•We evaluated data collection practices, statistical reporting, and data availability.•There was a high error rate in statistical reporting and overall low transparency.•Paths forward to advance our empirical understanding of VR effects are discussed.