Estimating Variance Components of Essay Ratings in a Complex Design

This paper discusses variables influencing written composition quality and how they can best be controlled to improve the reliability of assessment of writing ability. The study aimed to concentrate on the effects of different imposed time limits on student performance in written composition at grade 10 level using methods of generalizability theory; however, as the study progresses the major focus becomes methodological issues. Research studies and literature on this subject are reviewed before the design of the study is described. The sample of 320 students from Iowa and Newfoundland schools who were divided into eight groups (each group being assigned eight judges, who would assess the students' performance on two written topics under different time constraints) is presented. Scoring procedures are outlined. Discussed are the analytic methods of both Model 1 (a seven way analysis of variance model) and its limitations, and Model 2, which was adopted to analyze the variance for a confounded model incorporating elements of topic, time limit, and group. The results and limitations of the study are documented in the concluding section. Data analysis statistics are appended. (AEF)