Investigating the authenticity of computer- and paper-based ESL writing tests

•Switching to computer L2 writing tests from paper may increase test authenticity.•Test-takers’ typing skills has an effect on their computer-based writing scores.•Computer-based ESL writing tests may bias against low-income groups.•Increasing school computer access and typing proficiency may mitigate the bias. An assumption underlying replacing traditional paper-based (PB) writing tests with computer-based (CB) tests is that CB writing tests are more authentic than PB tests (Lessien, 2013). This study tested this assumption by examining the effect of test mode on the situational and interactive authenticity of English as a second language (ESL) writing assessments using an embedded correlational model of mixed-methods design. Sixty international ESL students enrolled at a university in the United States completed two writing tasks in the CB and PB modes. The participants then filled out cognitive processing questionnaires and participated in a post-test interview. All essays were scored by two raters both holistically and analytically. The results indicated higher authenticity for the CB test but also uncovered some concerns with the effect of typing skills on CB test performance. The findings have practical implications for test development, administration policies, and stakeholders in choosing delivery mode for ESL writing assessments.