Revisiting raters' accent familiarity in speaking tests : Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings
This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters who were all either familiar or unfamiliar with Brazilian-accente...
Gespeichert in:
| Veröffentlicht in: | Language testing 2024-04, Vol.41 (2), p.290-315 |
|---|---|
| Hauptverfasser: | , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 315 |
|---|---|
| container_issue | 2 |
| container_start_page | 290 |
| container_title | Language testing |
| container_volume | 41 |
| creator | Carey, Michael D. Szocs, Stefan |
| description | This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters
who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression
with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings (X2 = 88.005, df = 3, p < .0001), with presentation mode having a stronger
effect in the interaction than accent familiarity (X2 = 59.328, df = 1, p < .0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to
audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio- only or audio-visual modes of presentation should be evaluated through
research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-
COVID-19. [Author abstract] |
| doi_str_mv | 10.1177/02655322231200808 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_rmit_</sourceid><recordid>TN_cdi_proquest_journals_3015090585</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><informt_id>10.3316/aeipt.234296</informt_id><sage_id>10.1177_02655322231200808</sage_id><sourcerecordid>3015090585</sourcerecordid><originalsourceid>FETCH-LOGICAL-c367t-cc506c247093e81780bcb1c425865c70082cec73ffe1b8262c0e250dbda94a493</originalsourceid><addsrcrecordid>eNp9kc9u1DAQhyNEJZbCA3CzxIFTytiO7YQDEqrKH6lSpaqcLWcy2XXZxMF2F-0b8Zg4bCUOIE627O_7zdhTVa84XHBuzFsQWikphJBcALTQPqk2vDGmhq7RT6vNel-vwLPqeUr3ANBxqTfVz1s6-OSzn7csukwxvWEOkebMRjf5vXfR5yPzM0sLuW8rlinlxN6xq4MfaEZieecyWyKlYrnsw8ymMFBxSpzDwv7wefev1BzYoexcvz8yN46EmWGYStKO5uT7whWodFWqphfV2ej2iV4-rufV149Xd5ef6-ubT18uP1zXKLXJNaICjaIx0ElquWmhx55jI1SrFZryNQIJjSzVeN8KLRBIKBj6wXWNazp5Xr0-5S4xfH8oT7X34SHOpaSVwBV0oFpVKH6iMIaUIo12iX5y8Wg52HUg9q-BFOfi5CS3pT-p_xPen4Q4-Wwd-SXbRC7izvp5DL9PQ9zaIfg1RUquHykhG9Fp-QuKIaTv</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3015090585</pqid></control><display><type>article</type><title>Revisiting raters' accent familiarity in speaking tests : Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings</title><source>Access via SAGE</source><creator>Carey, Michael D. ; Szocs, Stefan</creator><creatorcontrib>Carey, Michael D. ; Szocs, Stefan</creatorcontrib><description>This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters
who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression
with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings (X2 = 88.005, df = 3, p < .0001), with presentation mode having a stronger
effect in the interaction than accent familiarity (X2 = 59.328, df = 1, p < .0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to
audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio- only or audio-visual modes of presentation should be evaluated through
research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-
COVID-19. [Author abstract]</description><identifier>ISSN: 0265-5322</identifier><identifier>EISSN: 1477-0946</identifier><identifier>DOI: 10.1177/02655322231200808</identifier><language>eng</language><publisher>London, England: SAGE Publications</publisher><subject>Accentuation ; Comprehensibility ; English (Second language) ; English as a second language tests ; Familiarity ; High stakes testing ; Intelligibility ; International English Language Testing System (IELTS) ; Language proficiency ; Language tests ; Pronunciation ; Rasch model ; Rater bias ; Speech skills ; Speech tests ; Spoken language ; Test of English as a Foreign Language Internet-Based Test (TOEFL iBT) ; Tok Pisin language ; Visual Aids</subject><ispartof>Language testing, 2024-04, Vol.41 (2), p.290-315</ispartof><rights>The Author(s) 2023</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c367t-cc506c247093e81780bcb1c425865c70082cec73ffe1b8262c0e250dbda94a493</citedby><cites>FETCH-LOGICAL-c367t-cc506c247093e81780bcb1c425865c70082cec73ffe1b8262c0e250dbda94a493</cites><orcidid>0000-0002-3117-9010 ; 0000-0002-1061-575X</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://journals.sagepub.com/doi/pdf/10.1177/02655322231200808$$EPDF$$P50$$Gsage$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://journals.sagepub.com/doi/10.1177/02655322231200808$$EHTML$$P50$$Gsage$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,21819,27924,27925,43621,43622</link.rule.ids></links><search><creatorcontrib>Carey, Michael D.</creatorcontrib><creatorcontrib>Szocs, Stefan</creatorcontrib><title>Revisiting raters' accent familiarity in speaking tests : Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings</title><title>Language testing</title><description>This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters
who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression
with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings (X2 = 88.005, df = 3, p < .0001), with presentation mode having a stronger
effect in the interaction than accent familiarity (X2 = 59.328, df = 1, p < .0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to
audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio- only or audio-visual modes of presentation should be evaluated through
research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-
COVID-19. [Author abstract]</description><subject>Accentuation</subject><subject>Comprehensibility</subject><subject>English (Second language)</subject><subject>English as a second language tests</subject><subject>Familiarity</subject><subject>High stakes testing</subject><subject>Intelligibility</subject><subject>International English Language Testing System (IELTS)</subject><subject>Language proficiency</subject><subject>Language tests</subject><subject>Pronunciation</subject><subject>Rasch model</subject><subject>Rater bias</subject><subject>Speech skills</subject><subject>Speech tests</subject><subject>Spoken language</subject><subject>Test of English as a Foreign Language Internet-Based Test (TOEFL iBT)</subject><subject>Tok Pisin language</subject><subject>Visual Aids</subject><issn>0265-5322</issn><issn>1477-0946</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>AFRWT</sourceid><recordid>eNp9kc9u1DAQhyNEJZbCA3CzxIFTytiO7YQDEqrKH6lSpaqcLWcy2XXZxMF2F-0b8Zg4bCUOIE627O_7zdhTVa84XHBuzFsQWikphJBcALTQPqk2vDGmhq7RT6vNel-vwLPqeUr3ANBxqTfVz1s6-OSzn7csukwxvWEOkebMRjf5vXfR5yPzM0sLuW8rlinlxN6xq4MfaEZieecyWyKlYrnsw8ymMFBxSpzDwv7wefev1BzYoexcvz8yN46EmWGYStKO5uT7whWodFWqphfV2ej2iV4-rufV149Xd5ef6-ubT18uP1zXKLXJNaICjaIx0ElquWmhx55jI1SrFZryNQIJjSzVeN8KLRBIKBj6wXWNazp5Xr0-5S4xfH8oT7X34SHOpaSVwBV0oFpVKH6iMIaUIo12iX5y8Wg52HUg9q-BFOfi5CS3pT-p_xPen4Q4-Wwd-SXbRC7izvp5DL9PQ9zaIfg1RUquHykhG9Fp-QuKIaTv</recordid><startdate>20240401</startdate><enddate>20240401</enddate><creator>Carey, Michael D.</creator><creator>Szocs, Stefan</creator><general>SAGE Publications</general><general>Sage Publications Ltd</general><scope>AFRWT</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7T9</scope><orcidid>https://orcid.org/0000-0002-3117-9010</orcidid><orcidid>https://orcid.org/0000-0002-1061-575X</orcidid></search><sort><creationdate>20240401</creationdate><title>Revisiting raters' accent familiarity in speaking tests : Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings</title><author>Carey, Michael D. ; Szocs, Stefan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c367t-cc506c247093e81780bcb1c425865c70082cec73ffe1b8262c0e250dbda94a493</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Accentuation</topic><topic>Comprehensibility</topic><topic>English (Second language)</topic><topic>English as a second language tests</topic><topic>Familiarity</topic><topic>High stakes testing</topic><topic>Intelligibility</topic><topic>International English Language Testing System (IELTS)</topic><topic>Language proficiency</topic><topic>Language tests</topic><topic>Pronunciation</topic><topic>Rasch model</topic><topic>Rater bias</topic><topic>Speech skills</topic><topic>Speech tests</topic><topic>Spoken language</topic><topic>Test of English as a Foreign Language Internet-Based Test (TOEFL iBT)</topic><topic>Tok Pisin language</topic><topic>Visual Aids</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Carey, Michael D.</creatorcontrib><creatorcontrib>Szocs, Stefan</creatorcontrib><collection>Sage Journals GOLD Open Access 2024</collection><collection>CrossRef</collection><collection>Linguistics and Language Behavior Abstracts (LLBA)</collection><jtitle>Language testing</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Carey, Michael D.</au><au>Szocs, Stefan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Revisiting raters' accent familiarity in speaking tests : Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings</atitle><jtitle>Language testing</jtitle><date>2024-04-01</date><risdate>2024</risdate><volume>41</volume><issue>2</issue><spage>290</spage><epage>315</epage><pages>290-315</pages><issn>0265-5322</issn><eissn>1477-0946</eissn><abstract>This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters
who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression
with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings (X2 = 88.005, df = 3, p < .0001), with presentation mode having a stronger
effect in the interaction than accent familiarity (X2 = 59.328, df = 1, p < .0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to
audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio- only or audio-visual modes of presentation should be evaluated through
research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-
COVID-19. [Author abstract]</abstract><cop>London, England</cop><pub>SAGE Publications</pub><doi>10.1177/02655322231200808</doi><tpages>26</tpages><orcidid>https://orcid.org/0000-0002-3117-9010</orcidid><orcidid>https://orcid.org/0000-0002-1061-575X</orcidid><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 0265-5322 |
| ispartof | Language testing, 2024-04, Vol.41 (2), p.290-315 |
| issn | 0265-5322 1477-0946 |
| language | eng |
| recordid | cdi_proquest_journals_3015090585 |
| source | Access via SAGE |
| subjects | Accentuation Comprehensibility English (Second language) English as a second language tests Familiarity High stakes testing Intelligibility International English Language Testing System (IELTS) Language proficiency Language tests Pronunciation Rasch model Rater bias Speech skills Speech tests Spoken language Test of English as a Foreign Language Internet-Based Test (TOEFL iBT) Tok Pisin language Visual Aids |
| title | Revisiting raters' accent familiarity in speaking tests : Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-02T01%3A33%3A54IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_rmit_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Revisiting%20raters'%20accent%20familiarity%20in%20speaking%20tests%20:%20Evidence%20that%20presentation%20mode%20interacts%20with%20accent%20familiarity%20to%20variably%20affect%20comprehensibility%20ratings&rft.jtitle=Language%20testing&rft.au=Carey,%20Michael%20D.&rft.date=2024-04-01&rft.volume=41&rft.issue=2&rft.spage=290&rft.epage=315&rft.pages=290-315&rft.issn=0265-5322&rft.eissn=1477-0946&rft_id=info:doi/10.1177/02655322231200808&rft_dat=%3Cproquest_rmit_%3E3015090585%3C/proquest_rmit_%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3015090585&rft_id=info:pmid/&rft_informt_id=10.3316/aeipt.234296&rft_sage_id=10.1177_02655322231200808&rfr_iscdi=true |