Validity of the Health Systems Science Examination: Relationship Between Examinee Performance and Time of Training

The health systems science (HSS) framework articulates systems-relevant topics that medical trainees must learn to be prepared for physician practice. As new HSS-related curricula are developed, measures demonstrating appropriate levels of reliability and validity are needed. The authors describe a collaborative effort between a consortium of medical schools and the National Board of Medical Examiners to create a multiple-choice HSS examination in the areas of evidence-based medicine/population health, patient safety, quality improvement, and teamwork. Fifteen schools administered the 100-question examination through 2 academic years a total of 1887 times to 1837 first-time takers. Total test score mean was 67% (SD 11%). Total test reliability as measured by coefficient α was .83. This examination differentiated between medical students who completed the examination before, during, and after relevant training/instruction. This new HSS examination can support and inform the efforts of institutions as they integrate HSS-related content into their curricula.