How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge
This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word erro...
Gespeichert in:
| Veröffentlicht in: | arXiv.org 2024-04 |
|---|---|
| Hauptverfasser: | , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | arXiv.org |
| container_volume | |
| creator | Mariana Arroyo Chavez Feanny, Molly Seita, Matthew Thompson, Bernard Delk, Keith Officer, Skyler Glasser, Abraham Kushalnagar, Raja Vogler, Christian |
| description | This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers. |
| doi_str_mv | 10.48550/arxiv.2404.10153 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2404_10153</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3040141830</sourcerecordid><originalsourceid>FETCH-LOGICAL-a520-6f3630f404316f53f4dfe6480faaf1f2ec4115a6336f28a58e23a68fed35e3b93</originalsourceid><addsrcrecordid>eNotkE9LAzEUxIMgWGo_gCcDnrcmeUmM3mSpVqiIsp48LM_2RVO2u2uy_fftXVtPA8OPYWYYu5BirJ0x4hrjLmzGSgs9lkIaOGEDBSAzp5U6Y6OUlkIIZW-UMTBgH9Nmy98TxcQnu5ZioHpOPK-aRAueY9uFpk68qfksbIgXVNEmpN67469rrEK358_UxTBP_I1WGGqOPP_GqqL6i87Zqccq0ehfh6x4mBT5NJu9PD7l97MMjRKZ9WBB-L4vSOsNeL3wZLUTHtFLr2iupTRoAaxXDo0jBWidpwUYgs9bGLLLY-xhednGsMK4L_8eKA8P9MTVkWhj87Om1JXLZh3rvlMJQguppQMBv1XxXcQ</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3040141830</pqid></control><display><type>article</type><title>How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge</title><source>arXiv.org</source><source>Free E- Journals</source><creator>Mariana Arroyo Chavez ; Feanny, Molly ; Seita, Matthew ; Thompson, Bernard ; Delk, Keith ; Officer, Skyler ; Glasser, Abraham ; Kushalnagar, Raja ; Vogler, Christian</creator><creatorcontrib>Mariana Arroyo Chavez ; Feanny, Molly ; Seita, Matthew ; Thompson, Bernard ; Delk, Keith ; Officer, Skyler ; Glasser, Abraham ; Kushalnagar, Raja ; Vogler, Christian</creatorcontrib><description>This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2404.10153</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Computer Science - Human-Computer Interaction ; Hearing ; Qualitative analysis ; Television ; User experience ; User satisfaction</subject><ispartof>arXiv.org, 2024-04</ispartof><rights>2024. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,776,780,881,27902</link.rule.ids><backlink>$$Uhttps://doi.org/10.1145/3613904.3641988$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.48550/arXiv.2404.10153$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Mariana Arroyo Chavez</creatorcontrib><creatorcontrib>Feanny, Molly</creatorcontrib><creatorcontrib>Seita, Matthew</creatorcontrib><creatorcontrib>Thompson, Bernard</creatorcontrib><creatorcontrib>Delk, Keith</creatorcontrib><creatorcontrib>Officer, Skyler</creatorcontrib><creatorcontrib>Glasser, Abraham</creatorcontrib><creatorcontrib>Kushalnagar, Raja</creatorcontrib><creatorcontrib>Vogler, Christian</creatorcontrib><title>How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge</title><title>arXiv.org</title><description>This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.</description><subject>Computer Science - Human-Computer Interaction</subject><subject>Hearing</subject><subject>Qualitative analysis</subject><subject>Television</subject><subject>User experience</subject><subject>User satisfaction</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>BENPR</sourceid><sourceid>GOX</sourceid><recordid>eNotkE9LAzEUxIMgWGo_gCcDnrcmeUmM3mSpVqiIsp48LM_2RVO2u2uy_fftXVtPA8OPYWYYu5BirJ0x4hrjLmzGSgs9lkIaOGEDBSAzp5U6Y6OUlkIIZW-UMTBgH9Nmy98TxcQnu5ZioHpOPK-aRAueY9uFpk68qfksbIgXVNEmpN67469rrEK358_UxTBP_I1WGGqOPP_GqqL6i87Zqccq0ehfh6x4mBT5NJu9PD7l97MMjRKZ9WBB-L4vSOsNeL3wZLUTHtFLr2iupTRoAaxXDo0jBWidpwUYgs9bGLLLY-xhednGsMK4L_8eKA8P9MTVkWhj87Om1JXLZh3rvlMJQguppQMBv1XxXcQ</recordid><startdate>20240415</startdate><enddate>20240415</enddate><creator>Mariana Arroyo Chavez</creator><creator>Feanny, Molly</creator><creator>Seita, Matthew</creator><creator>Thompson, Bernard</creator><creator>Delk, Keith</creator><creator>Officer, Skyler</creator><creator>Glasser, Abraham</creator><creator>Kushalnagar, Raja</creator><creator>Vogler, Christian</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20240415</creationdate><title>How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge</title><author>Mariana Arroyo Chavez ; Feanny, Molly ; Seita, Matthew ; Thompson, Bernard ; Delk, Keith ; Officer, Skyler ; Glasser, Abraham ; Kushalnagar, Raja ; Vogler, Christian</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a520-6f3630f404316f53f4dfe6480faaf1f2ec4115a6336f28a58e23a68fed35e3b93</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Human-Computer Interaction</topic><topic>Hearing</topic><topic>Qualitative analysis</topic><topic>Television</topic><topic>User experience</topic><topic>User satisfaction</topic><toplevel>online_resources</toplevel><creatorcontrib>Mariana Arroyo Chavez</creatorcontrib><creatorcontrib>Feanny, Molly</creatorcontrib><creatorcontrib>Seita, Matthew</creatorcontrib><creatorcontrib>Thompson, Bernard</creatorcontrib><creatorcontrib>Delk, Keith</creatorcontrib><creatorcontrib>Officer, Skyler</creatorcontrib><creatorcontrib>Glasser, Abraham</creatorcontrib><creatorcontrib>Kushalnagar, Raja</creatorcontrib><creatorcontrib>Vogler, Christian</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection (ProQuest)</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Mariana Arroyo Chavez</au><au>Feanny, Molly</au><au>Seita, Matthew</au><au>Thompson, Bernard</au><au>Delk, Keith</au><au>Officer, Skyler</au><au>Glasser, Abraham</au><au>Kushalnagar, Raja</au><au>Vogler, Christian</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge</atitle><jtitle>arXiv.org</jtitle><date>2024-04-15</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2404.10153</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | EISSN: 2331-8422 |
| ispartof | arXiv.org, 2024-04 |
| issn | 2331-8422 |
| language | eng |
| recordid | cdi_arxiv_primary_2404_10153 |
| source | arXiv.org; Free E- Journals |
| subjects | Computer Science - Human-Computer Interaction Hearing Qualitative analysis Television User experience User satisfaction |
| title | How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-13T08%3A17%3A16IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=How%20Users%20Experience%20Closed%20Captions%20on%20Live%20Television:%20Quality%20Metrics%20Remain%20a%20Challenge&rft.jtitle=arXiv.org&rft.au=Mariana%20Arroyo%20Chavez&rft.date=2024-04-15&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2404.10153&rft_dat=%3Cproquest_arxiv%3E3040141830%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3040141830&rft_id=info:pmid/&rfr_iscdi=true |