Using Eye‐Tracking Data as Part of the Validity Argument for Multiple‐Choice Questions: A Demonstration

Eye‐tracking technology can create a record of the location and duration of visual fixations as a test‐taker reads test questions. Although the cognitive process the test‐taker is using cannot be directly observed, eye‐tracking data can support inferences about these unobserved cognitive processes. This type of information has the potential to support improved test design and to contribute to an overall validity argument for the inferences and uses made based on test scores. Although several authors have referred to the potential usefulness of eye‐tracking data, there are relatively few published studies that provide examples of that use. In this paper, we report the results an eye‐tracking study designed to evaluate how the presence of the options in multiple‐choice questions impacts the way medical students responded to questions designed to evaluate clinical reasoning. Examples of the types of data that can be extracted are presented. We then discuss the implications of these results for evaluating the validity of inferences made based on the type of items used in this study.