Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from?

► The paper describes the development of a rating scale for diagnostic writing assessment. ► Weigle's (2002) steps of rating scale development are discussed. ► Several theoretical models are evaluated. Rating scales act as the de facto test construct in a writing assessment, although inevitably as a simplification of the construct ( North, 2003). However, it is often not reported how rating scales are constructed. Unless the underlying framework of a rating scale takes some account of linguistic theory and research in the definition of proficiency, the validity of the scale will be limited ( Lantolf & Frawley, 1985). In this paper, the decision-making process when designing a rating scale for diagnostic writing assessment is described. A number of considerations are discussed. These include the type of rating scale, who is going to use the rating scale, what the descriptors should look like and how the scores should be reported. The most central consideration is what the scale categories (or traits) should look like. While the literature on rating scales (e.g., Fulcher, 1987; McNamara, 2002) has called for scales to be theory-based, it is shown that none of the theories/models available for this purpose are sufficient by themselves for a scale of diagnostic writing. Several different theories and models were evaluated for the purpose of this paper, and the advantages and disadvantages of each are discussed.