An Application of Generalizability Theory and Many-Facet Rasch Measurement Using a Complex Problem-Solving Skills Assessment

This study describes the use of generalizability theory (GT) and many-facet Rasch measurement (MFRM) to evaluate psychometric properties of responses obtained from an assessment designed to measure complex problem-solving skills. The assessment revolved around the school activity of kickball. The task required of each student was to decide on a team T-shirt based on visual characteristics of the T-shirts and written descriptions of other T-shirt characteristics. Forty-four fourth-grade students, comprising the control group of a longitudinal research project to foster complex problem-solving skills of disadvantaged urban youth, participated in this study. Results indicate both measurement techniques agree on the relative magnitudes of variation among the facets but differ on how to handle the sources of variation.