Scientific Reasoning in Higher Education: Constructing and Evaluating the Criterion-Related Validity of an Assessment of Preservice Science Teachers' Competencies

The aim of this study was to develop a standardized test addressed to measure preservice science teachers' scientific reasoning skills, and to initially evaluate its psychometric properties. We constructed 123 multiple-choice items, using 259 students' conceptions to generate highly attractive multiple-choice response options. In an item response theory-based validation study (N = 2,247), we applied multiple regression analyses to test hypotheses based on groups with known attributes. As predicted, graduate students performed better than undergraduate students, and students who studied two natural science disciplines performed better than students who studied only one natural science discipline. In contrast to our initial hypothesis, preservice science teachers performed less well than a control group of natural sciences students. Remarkably, an interaction effect of the degree program (bachelor vs. master) and the qualification (natural sciences student vs. preservice teacher) was found, suggesting that preservice science teachers' learning opportunities to explicitly discuss and reflect on the inquiry process have a positive effect on the development of their scientific reasoning skills. We conclude that the evidence provides support for the criterion-based validity of our interpretation of the test scores as measures of scientific reasoning competencies.