The Reliability of Classroom Observations by School Personnel. Research Paper. MET Project

For many teachers, the classroom observation has been the only opportunity to receive direct feedback from another school professional. As such, it is an indispensable part of every teacher evaluation system. Yet it also requires a major time commitment from teachers, principals, and peer observers. To justify the investment of time and resources, a classroom observation should be both accurate and reliable. In this paper, the authors evaluate the accuracy and reliability of school personnel in performing classroom observations. The authors also examine different combinations of observers and lessons observed that produce reliability of 0.65 or above when using school personnel. They asked principals and peers in Hillsborough County, Florida, to watch and score videos of classroom teaching for 67 teacher-volunteers using videos of lessons captured during the 2011-12 school year. Each of 129 observers provided 24 scores on lessons provided to them, yielding more than 3,000 video scores for this analysis. The authors briefly summarize seven key findings: (1) Observers rarely used the top or bottom categories ("unsatisfactory" and "advanced") on the four-point observation instrument; (2) Compared to peer raters, administrators differentiated more among teachers; (3) Administrators rated their own teachers 0.1 points higher than administrators from other schools and 0.2 points higher than peers; (4) Although administrators scored their own teachers higher, their rankings were similar to the rankings produced by others outside their schools; (5) Allowing teachers to choose their own videos generated higher average scores. However, the relative ranking of teachers was preserved whether videos were chosen or not; (6) When an observer formed a positive (or negative) impression of a teacher in the first several videos, that impression tended to linger; and (7) There are a number of different ways to ensure reliability of 0.65 or above. The authors conclude by discussing the implications for the design of teacher evaluation systems in practice. (Contains 7 figures, 10 tables, and 21 footnotes.)