Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn

This study investigated the application of WriteToLearn on Chinese undergraduate English majors' essays in terms of its scoring ability and the accuracy of its error feedback. Participants were 163 second-year English majors from a university located in Sichuan province who wrote 326 essays from two writing prompts. Each paper was marked by four human raters as well as WriteToLearn. Many-facet Rasch measurement (MFRM) was conducted to calibrate WriteToLearn's rating performance in scoring the whole set of essays against those of four trained human raters. In addition, the accuracy of WriteToLearn's feedback on 60 randomly selected essays was compared with the feedback provided by human raters. The two main findings related to scoring were that: (1) WriteToLearn was more consistent but highly stringent when compared to the four trained human raters in scoring essays; and (2) WriteToLearn failed to score seven essays. In terms of error feedback, WriteToLearn had an overall precision and recall of 49% and 18.7% respectively. These figures did not meet the minimum threshold of 90% precision (set by Burstein, Chodorow, and Leacock, 2003) for it to be considered a reliable error detecting tool. Furthermore, it had difficulty in identifying errors made by Chinese undergraduate English majors in the use of articles, prepositions, word choice and expression.