Does difficulty-based item order matter in multiple-choice exams? (Empirical evidence from university students)

•Multiple-choice items are one of the most commonly used objective item formats.•Unfavorable item order in terms of item difficulty can be a source of random error.•Low, moderate and high achievers’ reaction to item-order is underresearched.•The effect of difficulty-based item order on test performance was examined.•Difficulty-based item order have no effect on test performance and item statistics. This empirical study aimed to investigate the impact of easy first vs. hard first ordering of the same items in a paper and-pencil multiple-choice exam on the performances of low, moderate, and high achiever examinees, as well as on the item statistics. Data were collected from 554 Turkish university students using two test forms, which included the same multiple-choice items ordered reversely, i.e. easy first vs. hard first. Tests included 26 multiple-choice items about the introductory unit of “Measurement and Assessment” course. The results suggested that sequencing the multiple-choice items in either direction from easy to hard or vice versa did not affect the test performances of the examinees no matter whether they are low, moderate or high achiever examinees. Finally, no statistically significant difference was observed between item statistics of both forms, i.e. the difficulty (p), discrimination (d), point biserial (r), and adjusted point biserial (adj. r) coefficients.