Application of an Empirical Bayes Enhancement of Mantel-Haenszel Differential Item Functioning Analysis to a Computerized Adaptive Test

This study used a simulation to investigate the applicability to computerized adaptive test data of a differential item functioning (DIF) analysis method developed by Zwick, Thayer, and Lewis. The approach involves an empirical Bayes (EB) enhancement of the popular Mantel-Haenszel (MH) DIF analysis method. Results showed the performance of the EB DIF approach to be quite promising, even in extremely small samples. In particular, the EB procedure was found to achieve roughly the same degree of stability for samples averaging 117 and 40 members in the two examinee groups as did the ordinary MH for samples averaging 240 in each of the two groups. Overall, the EB estimates tended to be closer to their target values than did the ordinary MH statistics in terms of root mean square residuals; the EB statistics were also more highly correlated with the target values than were the MH statistics. When combined with a loss-function-based decision rule, the EB method is better at detecting DIF than conventional approaches, but it has a higher Type I error rate.