Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods
Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for...
Gespeichert in:
| Veröffentlicht in: | Educational and psychological measurement 1999-06, Vol.59 (3), p.492-506 |
|---|---|
| Hauptverfasser: | , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 506 |
|---|---|
| container_issue | 3 |
| container_start_page | 492 |
| container_title | Educational and psychological measurement |
| container_volume | 59 |
| creator | Swartz, Carl W. Hooper, Stephen R. Montgomery, James W. Wakely, Melissa B. de Kruif, Renee E. L. Reed, Martha Brown, Timothy T. Levine, Melvin D. White, Kinnard P. |
| description | Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the simultaneous estimation of multiple sources of error variance to estimate the reliability of test scores. Using GT, two studies were conducted to investigate the impact of the number of raters and the type of decision (relative vs. absolute) on the reliability of writing scores. The results of both studies indicated that the reliability coefficients for writing scores decline as (a) the number of raters is reduced and (b) when absolute decisions rather than relative decisions are made. |
| doi_str_mv | 10.1177/00131649921970008 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_221559887</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ericid>EJ594286</ericid><sage_id>10.1177_00131649921970008</sage_id><sourcerecordid>41615778</sourcerecordid><originalsourceid>FETCH-LOGICAL-a408t-4e3f84a13bfc0fb54647ac2aa93868c096f2fafb5a9cbd32f5717402adc6f2883</originalsourceid><addsrcrecordid>eNp1kclKBDEURYMo2A4fILgI6rY0U1UlS9F2QhEccFm8TiV2pKxokhbalZ9u2nYCcRXCOTd53IfQBiW7lNb1HiGU00ooxaiqCSFyAQ1oWbKCSykX0WDGiyyIZbQS40M2iKB0gN5uo-vv8bHpTYDOvcLIdS5N8c3Y-DDFyeNhTO4RksFpbPCV6dyX4i2-Cy7N4tfaBxPxoQnuxbTYBv-IT3znclRj6Fu830M3zRfoPtxZ5sKksW_jGlqy0EWz_nmuotuj4c3BSXF-eXx6sH9egCAyFcJwKwVQPrKa2FEpKlGDZgCKy0pqoirLLGQASo9azmxZ01oQBq3OREq-irbm7z4F_zwxMTUPfhLyWLFhLDelpKyztP2fREUtGGeq5Nmic0sHH2MwtnkKuaIwbShpZtto_mwjZ3Y-X4aYW7ABeu3iT1AKURGWtc25lpvU33R4VirBZJXx7hxHuDe_Zvv323fiCqB0</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1474232953</pqid></control><display><type>article</type><title>Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods</title><source>SAGE Complete A-Z List</source><source>Periodicals Index Online</source><creator>Swartz, Carl W. ; Hooper, Stephen R. ; Montgomery, James W. ; Wakely, Melissa B. ; de Kruif, Renee E. L. ; Reed, Martha ; Brown, Timothy T. ; Levine, Melvin D. ; White, Kinnard P.</creator><creatorcontrib>Swartz, Carl W. ; Hooper, Stephen R. ; Montgomery, James W. ; Wakely, Melissa B. ; de Kruif, Renee E. L. ; Reed, Martha ; Brown, Timothy T. ; Levine, Melvin D. ; White, Kinnard P.</creatorcontrib><description>Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the simultaneous estimation of multiple sources of error variance to estimate the reliability of test scores. Using GT, two studies were conducted to investigate the impact of the number of raters and the type of decision (relative vs. absolute) on the reliability of writing scores. The results of both studies indicated that the reliability coefficients for writing scores decline as (a) the number of raters is reduced and (b) when absolute decisions rather than relative decisions are made.</description><identifier>ISSN: 0013-1644</identifier><identifier>EISSN: 1552-3888</identifier><identifier>DOI: 10.1177/00131649921970008</identifier><identifier>CODEN: EPMEAJ</identifier><language>eng</language><publisher>Thousand Oaks, CA: SAGE Publications</publisher><subject>Analytic Scoring ; Biological and medical sciences ; Educational evaluation ; Educational psychology ; Estimation (Mathematics) ; Fundamental and applied biological sciences. Psychology ; Generalizability Theory ; Holistic Evaluation ; Intermediate Grades ; Middle Schools ; Orientation. Evaluation ; Outcomes of Education ; Psychology. Psychoanalysis. Psychiatry ; Psychology. Psychophysiology ; Reliability ; Scores ; Scoring ; Test Theory ; Testing ; Tests ; Writing Tests</subject><ispartof>Educational and psychological measurement, 1999-06, Vol.59 (3), p.492-506</ispartof><rights>1999 INIST-CNRS</rights><rights>Copyright Sage Publications, Inc. Jun 1999</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-a408t-4e3f84a13bfc0fb54647ac2aa93868c096f2fafb5a9cbd32f5717402adc6f2883</citedby><cites>FETCH-LOGICAL-a408t-4e3f84a13bfc0fb54647ac2aa93868c096f2fafb5a9cbd32f5717402adc6f2883</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://journals.sagepub.com/doi/pdf/10.1177/00131649921970008$$EPDF$$P50$$Gsage$$H</linktopdf><linktohtml>$$Uhttps://journals.sagepub.com/doi/10.1177/00131649921970008$$EHTML$$P50$$Gsage$$H</linktohtml><link.rule.ids>314,776,780,21799,27848,27903,27904,43600,43601</link.rule.ids><backlink>$$Uhttp://eric.ed.gov/ERICWebPortal/detail?accno=EJ594286$$DView record in ERIC$$Hfree_for_read</backlink><backlink>$$Uhttp://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=1844602$$DView record in Pascal Francis$$Hfree_for_read</backlink></links><search><creatorcontrib>Swartz, Carl W.</creatorcontrib><creatorcontrib>Hooper, Stephen R.</creatorcontrib><creatorcontrib>Montgomery, James W.</creatorcontrib><creatorcontrib>Wakely, Melissa B.</creatorcontrib><creatorcontrib>de Kruif, Renee E. L.</creatorcontrib><creatorcontrib>Reed, Martha</creatorcontrib><creatorcontrib>Brown, Timothy T.</creatorcontrib><creatorcontrib>Levine, Melvin D.</creatorcontrib><creatorcontrib>White, Kinnard P.</creatorcontrib><title>Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods</title><title>Educational and psychological measurement</title><description>Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the simultaneous estimation of multiple sources of error variance to estimate the reliability of test scores. Using GT, two studies were conducted to investigate the impact of the number of raters and the type of decision (relative vs. absolute) on the reliability of writing scores. The results of both studies indicated that the reliability coefficients for writing scores decline as (a) the number of raters is reduced and (b) when absolute decisions rather than relative decisions are made.</description><subject>Analytic Scoring</subject><subject>Biological and medical sciences</subject><subject>Educational evaluation</subject><subject>Educational psychology</subject><subject>Estimation (Mathematics)</subject><subject>Fundamental and applied biological sciences. Psychology</subject><subject>Generalizability Theory</subject><subject>Holistic Evaluation</subject><subject>Intermediate Grades</subject><subject>Middle Schools</subject><subject>Orientation. Evaluation</subject><subject>Outcomes of Education</subject><subject>Psychology. Psychoanalysis. Psychiatry</subject><subject>Psychology. Psychophysiology</subject><subject>Reliability</subject><subject>Scores</subject><subject>Scoring</subject><subject>Test Theory</subject><subject>Testing</subject><subject>Tests</subject><subject>Writing Tests</subject><issn>0013-1644</issn><issn>1552-3888</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>1999</creationdate><recordtype>article</recordtype><sourceid>K30</sourceid><recordid>eNp1kclKBDEURYMo2A4fILgI6rY0U1UlS9F2QhEccFm8TiV2pKxokhbalZ9u2nYCcRXCOTd53IfQBiW7lNb1HiGU00ooxaiqCSFyAQ1oWbKCSykX0WDGiyyIZbQS40M2iKB0gN5uo-vv8bHpTYDOvcLIdS5N8c3Y-DDFyeNhTO4RksFpbPCV6dyX4i2-Cy7N4tfaBxPxoQnuxbTYBv-IT3znclRj6Fu830M3zRfoPtxZ5sKksW_jGlqy0EWz_nmuotuj4c3BSXF-eXx6sH9egCAyFcJwKwVQPrKa2FEpKlGDZgCKy0pqoirLLGQASo9azmxZ01oQBq3OREq-irbm7z4F_zwxMTUPfhLyWLFhLDelpKyztP2fREUtGGeq5Nmic0sHH2MwtnkKuaIwbShpZtto_mwjZ3Y-X4aYW7ABeu3iT1AKURGWtc25lpvU33R4VirBZJXx7hxHuDe_Zvv323fiCqB0</recordid><startdate>19990601</startdate><enddate>19990601</enddate><creator>Swartz, Carl W.</creator><creator>Hooper, Stephen R.</creator><creator>Montgomery, James W.</creator><creator>Wakely, Melissa B.</creator><creator>de Kruif, Renee E. L.</creator><creator>Reed, Martha</creator><creator>Brown, Timothy T.</creator><creator>Levine, Melvin D.</creator><creator>White, Kinnard P.</creator><general>SAGE Publications</general><general>Sage</general><general>Educational and Psychological Measurement, etc</general><general>SAGE PUBLICATIONS, INC</general><scope>7SW</scope><scope>BJH</scope><scope>BNH</scope><scope>BNI</scope><scope>BNJ</scope><scope>BNO</scope><scope>ERI</scope><scope>PET</scope><scope>REK</scope><scope>WWN</scope><scope>IQODW</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>EOLOZ</scope><scope>FKUCP</scope><scope>IOIBA</scope><scope>K30</scope><scope>PAAUG</scope><scope>PAWHS</scope><scope>PAWZZ</scope><scope>PAXOH</scope><scope>PBHAV</scope><scope>PBQSW</scope><scope>PBYQZ</scope><scope>PCIWU</scope><scope>PCMID</scope><scope>PCZJX</scope><scope>PDGRG</scope><scope>PDWWI</scope><scope>PETMR</scope><scope>PFVGT</scope><scope>PGXDX</scope><scope>PIHIL</scope><scope>PISVA</scope><scope>PJCTQ</scope><scope>PJTMS</scope><scope>PLCHJ</scope><scope>PMHAD</scope><scope>PNQDJ</scope><scope>POUND</scope><scope>PPLAD</scope><scope>PQAPC</scope><scope>PQCAN</scope><scope>PQCMW</scope><scope>PQEME</scope><scope>PQHKH</scope><scope>PQMID</scope><scope>PQNCT</scope><scope>PQNET</scope><scope>PQSCT</scope><scope>PQSET</scope><scope>PSVJG</scope><scope>PVMQY</scope><scope>PZGFC</scope></search><sort><creationdate>19990601</creationdate><title>Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods</title><author>Swartz, Carl W. ; Hooper, Stephen R. ; Montgomery, James W. ; Wakely, Melissa B. ; de Kruif, Renee E. L. ; Reed, Martha ; Brown, Timothy T. ; Levine, Melvin D. ; White, Kinnard P.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a408t-4e3f84a13bfc0fb54647ac2aa93868c096f2fafb5a9cbd32f5717402adc6f2883</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>1999</creationdate><topic>Analytic Scoring</topic><topic>Biological and medical sciences</topic><topic>Educational evaluation</topic><topic>Educational psychology</topic><topic>Estimation (Mathematics)</topic><topic>Fundamental and applied biological sciences. Psychology</topic><topic>Generalizability Theory</topic><topic>Holistic Evaluation</topic><topic>Intermediate Grades</topic><topic>Middle Schools</topic><topic>Orientation. Evaluation</topic><topic>Outcomes of Education</topic><topic>Psychology. Psychoanalysis. Psychiatry</topic><topic>Psychology. Psychophysiology</topic><topic>Reliability</topic><topic>Scores</topic><topic>Scoring</topic><topic>Test Theory</topic><topic>Testing</topic><topic>Tests</topic><topic>Writing Tests</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Swartz, Carl W.</creatorcontrib><creatorcontrib>Hooper, Stephen R.</creatorcontrib><creatorcontrib>Montgomery, James W.</creatorcontrib><creatorcontrib>Wakely, Melissa B.</creatorcontrib><creatorcontrib>de Kruif, Renee E. L.</creatorcontrib><creatorcontrib>Reed, Martha</creatorcontrib><creatorcontrib>Brown, Timothy T.</creatorcontrib><creatorcontrib>Levine, Melvin D.</creatorcontrib><creatorcontrib>White, Kinnard P.</creatorcontrib><collection>ERIC</collection><collection>ERIC (Ovid)</collection><collection>ERIC</collection><collection>ERIC</collection><collection>ERIC (Legacy Platform)</collection><collection>ERIC( SilverPlatter )</collection><collection>ERIC</collection><collection>ERIC PlusText (Legacy Platform)</collection><collection>Education Resources Information Center (ERIC)</collection><collection>ERIC</collection><collection>Pascal-Francis</collection><collection>CrossRef</collection><collection>Periodicals Index Online Segment 01</collection><collection>Periodicals Index Online Segment 04</collection><collection>Periodicals Index Online Segment 29</collection><collection>Periodicals Index Online</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - West</collection><collection>Primary Sources Access (Plan D) - International</collection><collection>Primary Sources Access & Build (Plan A) - MEA</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Midwest</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Northeast</collection><collection>Primary Sources Access (Plan D) - Southeast</collection><collection>Primary Sources Access (Plan D) - North Central</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Southeast</collection><collection>Primary Sources Access (Plan D) - South Central</collection><collection>Primary Sources Access & Build (Plan A) - UK / I</collection><collection>Primary Sources Access (Plan D) - Canada</collection><collection>Primary Sources Access (Plan D) - EMEALA</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - North Central</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - South Central</collection><collection>Primary Sources Access & Build (Plan A) - International</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - International</collection><collection>Primary Sources Access (Plan D) - West</collection><collection>Periodicals Index Online Segments 1-50</collection><collection>Primary Sources Access (Plan D) - APAC</collection><collection>Primary Sources Access (Plan D) - Midwest</collection><collection>Primary Sources Access (Plan D) - MEA</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Canada</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - UK / I</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - EMEALA</collection><collection>Primary Sources Access & Build (Plan A) - APAC</collection><collection>Primary Sources Access & Build (Plan A) - Canada</collection><collection>Primary Sources Access & Build (Plan A) - West</collection><collection>Primary Sources Access & Build (Plan A) - EMEALA</collection><collection>Primary Sources Access (Plan D) - Northeast</collection><collection>Primary Sources Access & Build (Plan A) - Midwest</collection><collection>Primary Sources Access & Build (Plan A) - North Central</collection><collection>Primary Sources Access & Build (Plan A) - Northeast</collection><collection>Primary Sources Access & Build (Plan A) - South Central</collection><collection>Primary Sources Access & Build (Plan A) - Southeast</collection><collection>Primary Sources Access (Plan D) - UK / I</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - APAC</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - MEA</collection><jtitle>Educational and psychological measurement</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Swartz, Carl W.</au><au>Hooper, Stephen R.</au><au>Montgomery, James W.</au><au>Wakely, Melissa B.</au><au>de Kruif, Renee E. L.</au><au>Reed, Martha</au><au>Brown, Timothy T.</au><au>Levine, Melvin D.</au><au>White, Kinnard P.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><ericid>EJ594286</ericid><atitle>Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods</atitle><jtitle>Educational and psychological measurement</jtitle><date>1999-06-01</date><risdate>1999</risdate><volume>59</volume><issue>3</issue><spage>492</spage><epage>506</epage><pages>492-506</pages><issn>0013-1644</issn><eissn>1552-3888</eissn><coden>EPMEAJ</coden><abstract>Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the simultaneous estimation of multiple sources of error variance to estimate the reliability of test scores. Using GT, two studies were conducted to investigate the impact of the number of raters and the type of decision (relative vs. absolute) on the reliability of writing scores. The results of both studies indicated that the reliability coefficients for writing scores decline as (a) the number of raters is reduced and (b) when absolute decisions rather than relative decisions are made.</abstract><cop>Thousand Oaks, CA</cop><pub>SAGE Publications</pub><doi>10.1177/00131649921970008</doi><tpages>15</tpages></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 0013-1644 |
| ispartof | Educational and psychological measurement, 1999-06, Vol.59 (3), p.492-506 |
| issn | 0013-1644 1552-3888 |
| language | eng |
| recordid | cdi_proquest_journals_221559887 |
| source | SAGE Complete A-Z List; Periodicals Index Online |
| subjects | Analytic Scoring Biological and medical sciences Educational evaluation Educational psychology Estimation (Mathematics) Fundamental and applied biological sciences. Psychology Generalizability Theory Holistic Evaluation Intermediate Grades Middle Schools Orientation. Evaluation Outcomes of Education Psychology. Psychoanalysis. Psychiatry Psychology. Psychophysiology Reliability Scores Scoring Test Theory Testing Tests Writing Tests |
| title | Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring Methods |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-27T07%3A21%3A01IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Using%20Generalizability%20Theory%20to%20Estimate%20the%20Reliability%20of%20Writing%20Scores%20Derived%20from%20Holistic%20and%20Analytical%20Scoring%20Methods&rft.jtitle=Educational%20and%20psychological%20measurement&rft.au=Swartz,%20Carl%20W.&rft.date=1999-06-01&rft.volume=59&rft.issue=3&rft.spage=492&rft.epage=506&rft.pages=492-506&rft.issn=0013-1644&rft.eissn=1552-3888&rft.coden=EPMEAJ&rft_id=info:doi/10.1177/00131649921970008&rft_dat=%3Cproquest_cross%3E41615778%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1474232953&rft_id=info:pmid/&rft_ericid=EJ594286&rft_sage_id=10.1177_00131649921970008&rfr_iscdi=true |