Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors

We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underli...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Educational measurement, issues and practice issues and practice, 2015-06, Vol.34 (2), p.34-46
Hauptverfasser: McCaffrey, Daniel F., Yuan, Kun, Savitsky, Terrance D., Lockwood, J. R., Edelen, Maria O.
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 46
container_issue 2
container_start_page 34
container_title Educational measurement, issues and practice
container_volume 34
creator McCaffrey, Daniel F.
Yuan, Kun
Savitsky, Terrance D.
Lockwood, J. R.
Edelen, Maria O.
description We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.
doi_str_mv 10.1111/emip.12061
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_1691828140</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ericid>EJ1065345</ericid><sourcerecordid>3729019321</sourcerecordid><originalsourceid>FETCH-LOGICAL-c3611-8d4a26cebdf726f4d9bac44f6b0b81caaa452b681900b089527b5bc6fd524853</originalsourceid><addsrcrecordid>eNp9kM1P4zAQxS0E0hbYy96RLHFDCngS20mOtJQv8SWWFQcOlu06YGjjMk4K_PebkF2OzGWkmd97o3mE_AK2D10duIVf7kPKJKyREeRcJFlZputkxPIMkn7wg2zG-MwYCFnmI_Lwp7Zh5dDXj_SynTd-pdHrxtHfDba2adFRX9PJXMeIISzotYkOV7rxoY79pnly9AZddLV1NFT0ttMinSIGjNtko9Lz6H7-61vk7nh6NzlNLq5PziaHF4nNJEBSzLhOpXVmVuWprPisNNpyXknDTAFWa81FamQBJWOGFaVIcyOMldVMpLwQ2RbZHWyXGF5bFxv1HFqsu4sKZAlFWgBnHbU3UBZD94ur1BL9QuOHAqb67FSfnfrMroN3BrgLxn6B03NgUmS8PwnD_s3P3cc3Tmp6eXbz3zMZND427v1Lo_FFyTzLhbq_OlFjkd0W4_GRguwv3bOKhg</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1691828140</pqid></control><display><type>article</type><title>Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors</title><source>Access via Wiley Online Library</source><source>EBSCOhost Education Source</source><creator>McCaffrey, Daniel F. ; Yuan, Kun ; Savitsky, Terrance D. ; Lockwood, J. R. ; Edelen, Maria O.</creator><creatorcontrib>McCaffrey, Daniel F. ; Yuan, Kun ; Savitsky, Terrance D. ; Lockwood, J. R. ; Edelen, Maria O.</creatorcontrib><description>We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.</description><identifier>ISSN: 0731-1745</identifier><identifier>EISSN: 1745-3992</identifier><identifier>DOI: 10.1111/emip.12061</identifier><language>eng</language><publisher>Washington: Blackwell Publishing Ltd</publisher><subject>CLASS-S ; classroom assessment scoring system ; Classroom management ; Classroom observation ; classroom observation protocols ; Classroom Observation Techniques ; Classroom Techniques ; Discriminant analysis ; Error of Measurement ; Factor Analysis ; Factor Structure ; Inferences ; Interrater Reliability ; Middle School Teachers ; Middle schools ; multilevel models ; Multivariate Analysis ; Observation ; Protocol Analysis ; Scores ; Teaching Methods ; Validity</subject><ispartof>Educational measurement, issues and practice, 2015-06, Vol.34 (2), p.34-46</ispartof><rights>2014 by the Educational Testing Service</rights><rights>Copyright Wiley Subscription Services, Inc. Summer 2015</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c3611-8d4a26cebdf726f4d9bac44f6b0b81caaa452b681900b089527b5bc6fd524853</citedby><cites>FETCH-LOGICAL-c3611-8d4a26cebdf726f4d9bac44f6b0b81caaa452b681900b089527b5bc6fd524853</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://onlinelibrary.wiley.com/doi/pdf/10.1111%2Femip.12061$$EPDF$$P50$$Gwiley$$H</linktopdf><linktohtml>$$Uhttps://onlinelibrary.wiley.com/doi/full/10.1111%2Femip.12061$$EHTML$$P50$$Gwiley$$H</linktohtml><link.rule.ids>314,780,784,1417,27924,27925,45574,45575</link.rule.ids><backlink>$$Uhttp://eric.ed.gov/ERICWebPortal/detail?accno=EJ1065345$$DView record in ERIC$$Hfree_for_read</backlink></links><search><creatorcontrib>McCaffrey, Daniel F.</creatorcontrib><creatorcontrib>Yuan, Kun</creatorcontrib><creatorcontrib>Savitsky, Terrance D.</creatorcontrib><creatorcontrib>Lockwood, J. R.</creatorcontrib><creatorcontrib>Edelen, Maria O.</creatorcontrib><title>Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors</title><title>Educational measurement, issues and practice</title><addtitle>Educational Measurement: Issues and Practice</addtitle><description>We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.</description><subject>CLASS-S</subject><subject>classroom assessment scoring system</subject><subject>Classroom management</subject><subject>Classroom observation</subject><subject>classroom observation protocols</subject><subject>Classroom Observation Techniques</subject><subject>Classroom Techniques</subject><subject>Discriminant analysis</subject><subject>Error of Measurement</subject><subject>Factor Analysis</subject><subject>Factor Structure</subject><subject>Inferences</subject><subject>Interrater Reliability</subject><subject>Middle School Teachers</subject><subject>Middle schools</subject><subject>multilevel models</subject><subject>Multivariate Analysis</subject><subject>Observation</subject><subject>Protocol Analysis</subject><subject>Scores</subject><subject>Teaching Methods</subject><subject>Validity</subject><issn>0731-1745</issn><issn>1745-3992</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2015</creationdate><recordtype>article</recordtype><recordid>eNp9kM1P4zAQxS0E0hbYy96RLHFDCngS20mOtJQv8SWWFQcOlu06YGjjMk4K_PebkF2OzGWkmd97o3mE_AK2D10duIVf7kPKJKyREeRcJFlZputkxPIMkn7wg2zG-MwYCFnmI_Lwp7Zh5dDXj_SynTd-pdHrxtHfDba2adFRX9PJXMeIISzotYkOV7rxoY79pnly9AZddLV1NFT0ttMinSIGjNtko9Lz6H7-61vk7nh6NzlNLq5PziaHF4nNJEBSzLhOpXVmVuWprPisNNpyXknDTAFWa81FamQBJWOGFaVIcyOMldVMpLwQ2RbZHWyXGF5bFxv1HFqsu4sKZAlFWgBnHbU3UBZD94ur1BL9QuOHAqb67FSfnfrMroN3BrgLxn6B03NgUmS8PwnD_s3P3cc3Tmp6eXbz3zMZND427v1Lo_FFyTzLhbq_OlFjkd0W4_GRguwv3bOKhg</recordid><startdate>20150601</startdate><enddate>20150601</enddate><creator>McCaffrey, Daniel F.</creator><creator>Yuan, Kun</creator><creator>Savitsky, Terrance D.</creator><creator>Lockwood, J. R.</creator><creator>Edelen, Maria O.</creator><general>Blackwell Publishing Ltd</general><general>Wiley-Blackwell</general><general>Wiley Subscription Services, Inc</general><scope>BSCLL</scope><scope>7SW</scope><scope>BJH</scope><scope>BNH</scope><scope>BNI</scope><scope>BNJ</scope><scope>BNO</scope><scope>ERI</scope><scope>PET</scope><scope>REK</scope><scope>WWN</scope><scope>AAYXX</scope><scope>CITATION</scope></search><sort><creationdate>20150601</creationdate><title>Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors</title><author>McCaffrey, Daniel F. ; Yuan, Kun ; Savitsky, Terrance D. ; Lockwood, J. R. ; Edelen, Maria O.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c3611-8d4a26cebdf726f4d9bac44f6b0b81caaa452b681900b089527b5bc6fd524853</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2015</creationdate><topic>CLASS-S</topic><topic>classroom assessment scoring system</topic><topic>Classroom management</topic><topic>Classroom observation</topic><topic>classroom observation protocols</topic><topic>Classroom Observation Techniques</topic><topic>Classroom Techniques</topic><topic>Discriminant analysis</topic><topic>Error of Measurement</topic><topic>Factor Analysis</topic><topic>Factor Structure</topic><topic>Inferences</topic><topic>Interrater Reliability</topic><topic>Middle School Teachers</topic><topic>Middle schools</topic><topic>multilevel models</topic><topic>Multivariate Analysis</topic><topic>Observation</topic><topic>Protocol Analysis</topic><topic>Scores</topic><topic>Teaching Methods</topic><topic>Validity</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>McCaffrey, Daniel F.</creatorcontrib><creatorcontrib>Yuan, Kun</creatorcontrib><creatorcontrib>Savitsky, Terrance D.</creatorcontrib><creatorcontrib>Lockwood, J. R.</creatorcontrib><creatorcontrib>Edelen, Maria O.</creatorcontrib><collection>Istex</collection><collection>ERIC</collection><collection>ERIC (Ovid)</collection><collection>ERIC</collection><collection>ERIC</collection><collection>ERIC (Legacy Platform)</collection><collection>ERIC( SilverPlatter )</collection><collection>ERIC</collection><collection>ERIC PlusText (Legacy Platform)</collection><collection>Education Resources Information Center (ERIC)</collection><collection>ERIC</collection><collection>CrossRef</collection><jtitle>Educational measurement, issues and practice</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>McCaffrey, Daniel F.</au><au>Yuan, Kun</au><au>Savitsky, Terrance D.</au><au>Lockwood, J. R.</au><au>Edelen, Maria O.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><ericid>EJ1065345</ericid><atitle>Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors</atitle><jtitle>Educational measurement, issues and practice</jtitle><addtitle>Educational Measurement: Issues and Practice</addtitle><date>2015-06-01</date><risdate>2015</risdate><volume>34</volume><issue>2</issue><spage>34</spage><epage>46</epage><pages>34-46</pages><issn>0731-1745</issn><eissn>1745-3992</eissn><abstract>We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.</abstract><cop>Washington</cop><pub>Blackwell Publishing Ltd</pub><doi>10.1111/emip.12061</doi><tpages>13</tpages></addata></record>
fulltext fulltext
identifier ISSN: 0731-1745
ispartof Educational measurement, issues and practice, 2015-06, Vol.34 (2), p.34-46
issn 0731-1745
1745-3992
language eng
recordid cdi_proquest_journals_1691828140
source Access via Wiley Online Library; EBSCOhost Education Source
subjects CLASS-S
classroom assessment scoring system
Classroom management
Classroom observation
classroom observation protocols
Classroom Observation Techniques
Classroom Techniques
Discriminant analysis
Error of Measurement
Factor Analysis
Factor Structure
Inferences
Interrater Reliability
Middle School Teachers
Middle schools
multilevel models
Multivariate Analysis
Observation
Protocol Analysis
Scores
Teaching Methods
Validity
title Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T12%3A09%3A46IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Uncovering%20Multivariate%20Structure%20in%20Classroom%20Observations%20in%20the%20Presence%20of%20Rater%20Errors&rft.jtitle=Educational%20measurement,%20issues%20and%20practice&rft.au=McCaffrey,%20Daniel%20F.&rft.date=2015-06-01&rft.volume=34&rft.issue=2&rft.spage=34&rft.epage=46&rft.pages=34-46&rft.issn=0731-1745&rft.eissn=1745-3992&rft_id=info:doi/10.1111/emip.12061&rft_dat=%3Cproquest_cross%3E3729019321%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1691828140&rft_id=info:pmid/&rft_ericid=EJ1065345&rfr_iscdi=true