Facilitating learners' self‐assessment during formative writing tasks using writing analytics toolkit

Background Learners' writing skills are critical to their academic and professional development. Previous studies have shown that learners' self‐assessment during writing is essential for assessing their writing products and monitoring their writing processes. However, conducting practical self‐assessments of writing remains challenging for learners without help, such as formative feedback. Objectives To facilitate learners' self‐assessment in writing, we developed a writing analytics toolkit and used data visualisation and cutting‐edge machine learning technology that provides real‐time and formative feedback to learners. Methods To investigate whether our newly‐developed tool affects the accuracy and process of learners' self‐assessment, we conducted a lab study. We assigned 59 learners to complete writing (2 h) and revising (1 h) tasks. During the revision stage, we randomly assigned the learners to two groups: one group used the writing analytics toolkit while the second group was not granted access to the toolkit. Learners' self‐assessment accuracy and process of self‐assessment were compared between the two groups. Results In our study, we found the toolkit helped learners in the experimental group improve the self‐assessment accuracy of their writing products compared to the learners in the control group. In addition, we also found that the affordances of the toolkit affected the learners' self‐assessment process, and poor design affordances may have prevented the learners from reflecting by themselves. Conclusions Together, our empirical study shed light on the design of future writing analytics tools which aim at improving learners' self‐assessment during formative writing processes. Lay Description What is already known about this topic Learners' self‐assessment during writing is essential for assessing their writing products and monitoring their writing processes. Formative feedback is vital for improving learners' self‐assessment capabilities. Writing analytics tools offer feedback to learners, however, how these tools facilitate self‐assessment remains under investigated. What this paper adds We developed a writing analytics toolkit and used data visualisation and cutting‐edge artificial intelligence technology that provides real‐time and formative feedback to learners. We found the toolkit helped learners improve the self‐assessment accuracy of their writing products. We also found that the affordances of the toolkit affected the learners' se