Effect of a Representative Sample of Internally Calibrated Mental Effort and Polytomously Scored Data on Representing Cognitive Efficiency

Practice examinations developed at the ACS Examinations Institute ask students to self-report mental effort when answering items. This self-reported mental effort together with performance can be represented in the form of a cognitive efficiency graph for each student giving information on the utilization of cognitive resources and content knowledge. When performance and mental effort used in the cognitive efficiency graph for a student is based on dichotomously scored data and raw reported mental effort, limitations arise. For example, it might be difficult to see a greater differentiation among students, and instructors might be interpreting student cognitive efficiency in certain content areas incorrectly. This study utilizes organic chemistry practice exam data to report an internal calibration method where a student’s mental effort is calibrated using their average reported mental effort across all items. Additionally, this study describes the benefits of using partial-credit assignment to provide greater differentiation in student performance. Results suggest that a more accurate cognitive efficiency graph can be obtained leading to a better assessment of student utilization of cognitive resources for different content areas. These results can have implications for how similar data on other ACS exams or similar data being used by researchers and practitioners can be analyzed to better represent cognitive efficiency.