Elo Rating Algorithm for the Purpose of Measuring Task Difficulty in Online Learning Environments

The Elo rating algorithm, developed for the purpose of measuring player strength in chess tournaments,has also found application in the context of educational research and has been used for the purpose ofmeasuring both learner ability and task difficulty. The quality of the estimations performed by the Elo ratingalgorithm has already been subject to research, and has been shown to deliver accurate estimations in bothlow and high-stake testing situations. However, little is known about the performance of the Elo algorithmin the context of learning environments where multiple attempts are allowed, feedback is provided, and thelearning process spans several weeks or even months. This study develops the topic of Elo algorithm usein an educational context and examines its performance on real data from an online learning environmentwhere multiple attempts were allowed, and feedback was provided after each attempt. Its performance interms of stability of the estimation results in two analyzed periods for two groups of learners with differentinitial levels of knowledge are compared with alternative difficulty estimation methods: proportion correctand learner feedback. According to the results, the Elo rating algorithm outperforms both proportion correctand learning feedback. It delivers stable difficulty estimations, with correlations in the range 0.87–0.92 forthe group of beginners and 0.72–0.84 for the group of experienced learners.