Invalid Methods and False Answers: Physics Education Research and the Use of GREs

Finding good educational policies requires sound estimates of their potential effects. Methods for making such estimates, i.e., finding causal estimands, have made great progress in the last few decades. Nevertheless, serious errors in causal reasoning have been found previously in papers in a leading physics education journal, Physical Review Physics Education Research. Here I examine three papers from that journal that present explicit methods of causal inference. The methods offered involve major errors, including errors in identifying causal mediation, choosing variables to control for, and imputing missing data. The erroneous methods lead to major underestimation of the predictive power of Graduate Record Exams.