Comparative evaluation of automated scoring of syntactic competence of non-native speakers

Syntactic competence, especially the ability to use a wide range of sophisticated grammatical expressions, represents an important aspect of communicative acumen. This paper explores the question of how to best evaluate the syntactic competence of non-native speakers in an automated way. Using spoken responses of test takers participating in an English practice assessment, three classes of grammatical features – features based on n-grams of part-of-speech tags (POS), features based on various clause types, and features based on various phrases – are compared in an end-to-end assessment system. Feature correlations with human proficiency scores show that POS features and phrase features exhibit the highest correlations with human scores. Including these three classes of grammar features in a baseline scoring model that measures various aspects of spoken proficiency excluding aspects of grammar, we find substantial increases in agreement between machine and human scores. Finally, we discuss the broader implications of our results on the design of automatic scoring systems for spoken language. •Comparison of various grammar features in context of automated speech scoring.•Features related to shorter text spans show higher correlations with human scores.•Adding grammar features to baseline model improves correlation with human scores.•Computing features on concatenation of spoken responses improves performance.