Estimating within-group interrater reliability with and without response bias
Presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manusc...
Gespeichert in:
| Veröffentlicht in: | Journal of applied psychology 1984-02, Vol.69 (1), p.85-98 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 98 |
|---|---|
| container_issue | 1 |
| container_start_page | 85 |
| container_title | Journal of applied psychology |
| container_volume | 69 |
| creator | James, Lawrence R Demaree, Robert G Wolf, Gerrit |
| description | Presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower level manager, or a team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For such situations, these procedures furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. The proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency and social desirability) on estimates of interrater reliability. (49 ref) |
| doi_str_mv | 10.1037/0021-9010.69.1.85 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_614331526</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>614331526</sourcerecordid><originalsourceid>FETCH-LOGICAL-a336t-66ef3fe80e1706ffcea704174c1876fad155c03629134a2306ba967c3e2b4e9f3</originalsourceid><addsrcrecordid>eNp9kM1OwzAQhC0EEqXwANwi4JrijRPHPqKq_EggLnC2Nq5dXIUk2I5Q356krcoJLl5b8-2sdwi5BDoDyspbSjNIJR2eXM5gJoojMgHJZAqiyI_J5KCfkrMQ1pRCziSdkJdFiO4To2tWybeLH65JV77tu8Q10XiPw5F4UzusXO3iZssk2Cy3l7aPgxi6tgkmqRyGc3JisQ7mYl-n5P1-8TZ_TJ9fH57md88pMsZjyrmxzBpBDZSUW6sNljSHMtcgSm5xCUWhKeOZBJZjxiivUPJSM5NVuZGWTcnVzrfz7VdvQlTrtvfNMFLxYTEGRcb_gzJgkgkp6QBd_wVBJmkuSmAjBTtK-zYEb6zq_JCa3yigasxfjfmqMV_FpQIliqHnZu-MQWNtPTbahUOjELKQwH8_gB2qLmw0-uh0bYLCrj6Y_QBDZ4_o</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>614331526</pqid></control><display><type>article</type><title>Estimating within-group interrater reliability with and without response bias</title><source>EBSCOhost APA PsycARTICLES</source><source>Periodicals Index Online</source><source>Applied Social Sciences Index & Abstracts (ASSIA)</source><creator>James, Lawrence R ; Demaree, Robert G ; Wolf, Gerrit</creator><creatorcontrib>James, Lawrence R ; Demaree, Robert G ; Wolf, Gerrit</creatorcontrib><description>Presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower level manager, or a team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For such situations, these procedures furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. The proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency and social desirability) on estimates of interrater reliability. (49 ref)</description><identifier>ISSN: 0021-9010</identifier><identifier>EISSN: 1939-1854</identifier><identifier>DOI: 10.1037/0021-9010.69.1.85</identifier><identifier>CODEN: JAPGBP</identifier><language>eng</language><publisher>Washington, DC: American Psychological Association</publisher><subject>Bias ; Biological and medical sciences ; Estimation ; Fundamental and applied biological sciences. Psychology ; Human ; Interrater Reliability ; Methodology ; Miscellaneous ; Psychology. Psychoanalysis. Psychiatry ; Psychology. Psychophysiology ; Ratings ; Reliability ; Response Bias ; Responses ; Social psychology ; Statistical analysis ; Studies</subject><ispartof>Journal of applied psychology, 1984-02, Vol.69 (1), p.85-98</ispartof><rights>1984 American Psychological Association</rights><rights>1985 INIST-CNRS</rights><rights>Copyright American Psychological Association Feb 1984</rights><rights>1984, American Psychological Association</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27869,27924,27925,30999</link.rule.ids><backlink>$$Uhttp://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=8895916$$DView record in Pascal Francis$$Hfree_for_read</backlink></links><search><creatorcontrib>James, Lawrence R</creatorcontrib><creatorcontrib>Demaree, Robert G</creatorcontrib><creatorcontrib>Wolf, Gerrit</creatorcontrib><title>Estimating within-group interrater reliability with and without response bias</title><title>Journal of applied psychology</title><description>Presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower level manager, or a team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For such situations, these procedures furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. The proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency and social desirability) on estimates of interrater reliability. (49 ref)</description><subject>Bias</subject><subject>Biological and medical sciences</subject><subject>Estimation</subject><subject>Fundamental and applied biological sciences. Psychology</subject><subject>Human</subject><subject>Interrater Reliability</subject><subject>Methodology</subject><subject>Miscellaneous</subject><subject>Psychology. Psychoanalysis. Psychiatry</subject><subject>Psychology. Psychophysiology</subject><subject>Ratings</subject><subject>Reliability</subject><subject>Response Bias</subject><subject>Responses</subject><subject>Social psychology</subject><subject>Statistical analysis</subject><subject>Studies</subject><issn>0021-9010</issn><issn>1939-1854</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>1984</creationdate><recordtype>article</recordtype><sourceid>K30</sourceid><sourceid>7QJ</sourceid><recordid>eNp9kM1OwzAQhC0EEqXwANwi4JrijRPHPqKq_EggLnC2Nq5dXIUk2I5Q356krcoJLl5b8-2sdwi5BDoDyspbSjNIJR2eXM5gJoojMgHJZAqiyI_J5KCfkrMQ1pRCziSdkJdFiO4To2tWybeLH65JV77tu8Q10XiPw5F4UzusXO3iZssk2Cy3l7aPgxi6tgkmqRyGc3JisQ7mYl-n5P1-8TZ_TJ9fH57md88pMsZjyrmxzBpBDZSUW6sNljSHMtcgSm5xCUWhKeOZBJZjxiivUPJSM5NVuZGWTcnVzrfz7VdvQlTrtvfNMFLxYTEGRcb_gzJgkgkp6QBd_wVBJmkuSmAjBTtK-zYEb6zq_JCa3yigasxfjfmqMV_FpQIliqHnZu-MQWNtPTbahUOjELKQwH8_gB2qLmw0-uh0bYLCrj6Y_QBDZ4_o</recordid><startdate>198402</startdate><enddate>198402</enddate><creator>James, Lawrence R</creator><creator>Demaree, Robert G</creator><creator>Wolf, Gerrit</creator><general>American Psychological Association</general><general>American Psychological Association, etc</general><scope>IQODW</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>EOLOZ</scope><scope>FKUCP</scope><scope>IOIBA</scope><scope>K30</scope><scope>PAAUG</scope><scope>PAWHS</scope><scope>PAWZZ</scope><scope>PAXOH</scope><scope>PBHAV</scope><scope>PBQSW</scope><scope>PBYQZ</scope><scope>PCIWU</scope><scope>PCMID</scope><scope>PCZJX</scope><scope>PDGRG</scope><scope>PDWWI</scope><scope>PETMR</scope><scope>PFVGT</scope><scope>PGXDX</scope><scope>PIHIL</scope><scope>PISVA</scope><scope>PJCTQ</scope><scope>PJTMS</scope><scope>PLCHJ</scope><scope>PMHAD</scope><scope>PNQDJ</scope><scope>POUND</scope><scope>PPLAD</scope><scope>PQAPC</scope><scope>PQCAN</scope><scope>PQCMW</scope><scope>PQEME</scope><scope>PQHKH</scope><scope>PQMID</scope><scope>PQNCT</scope><scope>PQNET</scope><scope>PQSCT</scope><scope>PQSET</scope><scope>PSVJG</scope><scope>PVMQY</scope><scope>PZGFC</scope><scope>7QJ</scope><scope>8BJ</scope><scope>FQK</scope><scope>JBE</scope><scope>7RZ</scope><scope>PSYQQ</scope></search><sort><creationdate>198402</creationdate><title>Estimating within-group interrater reliability with and without response bias</title><author>James, Lawrence R ; Demaree, Robert G ; Wolf, Gerrit</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a336t-66ef3fe80e1706ffcea704174c1876fad155c03629134a2306ba967c3e2b4e9f3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>1984</creationdate><topic>Bias</topic><topic>Biological and medical sciences</topic><topic>Estimation</topic><topic>Fundamental and applied biological sciences. Psychology</topic><topic>Human</topic><topic>Interrater Reliability</topic><topic>Methodology</topic><topic>Miscellaneous</topic><topic>Psychology. Psychoanalysis. Psychiatry</topic><topic>Psychology. Psychophysiology</topic><topic>Ratings</topic><topic>Reliability</topic><topic>Response Bias</topic><topic>Responses</topic><topic>Social psychology</topic><topic>Statistical analysis</topic><topic>Studies</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>James, Lawrence R</creatorcontrib><creatorcontrib>Demaree, Robert G</creatorcontrib><creatorcontrib>Wolf, Gerrit</creatorcontrib><collection>Pascal-Francis</collection><collection>CrossRef</collection><collection>Periodicals Index Online Segment 01</collection><collection>Periodicals Index Online Segment 04</collection><collection>Periodicals Index Online Segment 29</collection><collection>Periodicals Index Online</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - West</collection><collection>Primary Sources Access (Plan D) - International</collection><collection>Primary Sources Access & Build (Plan A) - MEA</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Midwest</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Northeast</collection><collection>Primary Sources Access (Plan D) - Southeast</collection><collection>Primary Sources Access (Plan D) - North Central</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Southeast</collection><collection>Primary Sources Access (Plan D) - South Central</collection><collection>Primary Sources Access & Build (Plan A) - UK / I</collection><collection>Primary Sources Access (Plan D) - Canada</collection><collection>Primary Sources Access (Plan D) - EMEALA</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - North Central</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - South Central</collection><collection>Primary Sources Access & Build (Plan A) - International</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - International</collection><collection>Primary Sources Access (Plan D) - West</collection><collection>Periodicals Index Online Segments 1-50</collection><collection>Primary Sources Access (Plan D) - APAC</collection><collection>Primary Sources Access (Plan D) - Midwest</collection><collection>Primary Sources Access (Plan D) - MEA</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - Canada</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - UK / I</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - EMEALA</collection><collection>Primary Sources Access & Build (Plan A) - APAC</collection><collection>Primary Sources Access & Build (Plan A) - Canada</collection><collection>Primary Sources Access & Build (Plan A) - West</collection><collection>Primary Sources Access & Build (Plan A) - EMEALA</collection><collection>Primary Sources Access (Plan D) - Northeast</collection><collection>Primary Sources Access & Build (Plan A) - Midwest</collection><collection>Primary Sources Access & Build (Plan A) - North Central</collection><collection>Primary Sources Access & Build (Plan A) - Northeast</collection><collection>Primary Sources Access & Build (Plan A) - South Central</collection><collection>Primary Sources Access & Build (Plan A) - Southeast</collection><collection>Primary Sources Access (Plan D) - UK / I</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - APAC</collection><collection>Primary Sources Access—Foundation Edition (Plan E) - MEA</collection><collection>Applied Social Sciences Index & Abstracts (ASSIA)</collection><collection>International Bibliography of the Social Sciences (IBSS)</collection><collection>International Bibliography of the Social Sciences</collection><collection>International Bibliography of the Social Sciences</collection><collection>Access via APA PsycArticles® (ProQuest)</collection><collection>ProQuest One Psychology</collection><jtitle>Journal of applied psychology</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>James, Lawrence R</au><au>Demaree, Robert G</au><au>Wolf, Gerrit</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Estimating within-group interrater reliability with and without response bias</atitle><jtitle>Journal of applied psychology</jtitle><date>1984-02</date><risdate>1984</risdate><volume>69</volume><issue>1</issue><spage>85</spage><epage>98</epage><pages>85-98</pages><issn>0021-9010</issn><eissn>1939-1854</eissn><coden>JAPGBP</coden><abstract>Presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower level manager, or a team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For such situations, these procedures furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. The proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency and social desirability) on estimates of interrater reliability. (49 ref)</abstract><cop>Washington, DC</cop><pub>American Psychological Association</pub><doi>10.1037/0021-9010.69.1.85</doi><tpages>14</tpages></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 0021-9010 |
| ispartof | Journal of applied psychology, 1984-02, Vol.69 (1), p.85-98 |
| issn | 0021-9010 1939-1854 |
| language | eng |
| recordid | cdi_proquest_journals_614331526 |
| source | EBSCOhost APA PsycARTICLES; Periodicals Index Online; Applied Social Sciences Index & Abstracts (ASSIA) |
| subjects | Bias Biological and medical sciences Estimation Fundamental and applied biological sciences. Psychology Human Interrater Reliability Methodology Miscellaneous Psychology. Psychoanalysis. Psychiatry Psychology. Psychophysiology Ratings Reliability Response Bias Responses Social psychology Statistical analysis Studies |
| title | Estimating within-group interrater reliability with and without response bias |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-02T06%3A54%3A56IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Estimating%20within-group%20interrater%20reliability%20with%20and%20without%20response%20bias&rft.jtitle=Journal%20of%20applied%20psychology&rft.au=James,%20Lawrence%20R&rft.date=1984-02&rft.volume=69&rft.issue=1&rft.spage=85&rft.epage=98&rft.pages=85-98&rft.issn=0021-9010&rft.eissn=1939-1854&rft.coden=JAPGBP&rft_id=info:doi/10.1037/0021-9010.69.1.85&rft_dat=%3Cproquest_cross%3E614331526%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=614331526&rft_id=info:pmid/&rfr_iscdi=true |