Student perception and usage of an automated programming assessment tool

•We empirical evaluate Mooshak, an automated computer programming assessment system.•We provide a highly detailed analysis of student usage of Mooshak.•The usage of Mooshak was adequate in relation to students’ own testing.•Mooshak’s feedback should be richer to improve its acceptance among students.•We found no statistical evidence to claim Mooshak reduced the dropout rate. Automated assessment systems are gaining popularity within computer programming courses. In this paper we perform an empirical evaluation of Mooshak, an online judge that verifies program correctness, in order to determine its usefulness in classroom settings. In particular, we provide a detailed study on how students use the tool, analyze their opinions and critiques about it, and measure other features like its capability to reduce dropout rates. The experience was carried out within a course on algorithm design and analysis where we collected information through several questionnaires and data generated by the tool during the course. Among the main findings we highlight: (1) the usage of the tool was adequate in relation to students’ own testing; (2) its feedback needs to be richer in order to improve its acceptance among students; and (3) there was no statistical evidence to claim Mooshak reduced the dropout rate.