Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification
Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects...
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | Keleg, Amr Magdy, Walid |
| description | Automatic Arabic Dialect Identification (ADI) of text has gained great
popularity since it was introduced in the early 2010s. Multiple datasets were
developed, and yearly shared tasks have been running since 2018. However, ADI
systems are reported to fail in distinguishing between the micro-dialects of
Arabic. We argue that the currently adopted framing of the ADI task as a
single-label classification problem is one of the main reasons for that. We
highlight the limitation of the incompleteness of the Dialect labels and
demonstrate how it impacts the evaluation of ADI systems. A manual error
analysis for the predictions of an ADI, performed by 7 native speakers of
different Arabic dialects, revealed that $\approx$ 66% of the validated errors
are not true errors. Consequently, we propose framing ADI as a multi-label
classification task and give recommendations for designing new ADI datasets. |
| doi_str_mv | 10.48550/arxiv.2310.13661 |
| format | Article |
| fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2310_13661</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2310_13661</sourcerecordid><originalsourceid>FETCH-LOGICAL-a671-b1c44f7d5494af49cace09a11259adf6d672aff40f418a3407026b06aa6c1a7d3</originalsourceid><addsrcrecordid>eNo9j8tOwzAURL1hgQofwAr_QIod39gNuyq8KkVi0S7YRTd-oCu5LnJcRP8eCKirkWZ0RjqM3UixhFXTiDvMX_S5rNVPIZXW8pK9rTOOZPkDYfS28I3zqVAgi4UOiR-T85lvbT4WSqd73tOeyjxN_BD4ltJ79FXE0UfeRZymM3rFLgLGyV__54Ltnh533UvVvz5vunVfoTayGqUFCMY10AIGaC1aL1qUsm5adEE7bWoMAUQAuUIFwohaj0IjaivROLVgt3-3s9rwkWmP-TT8Kg6zovoGScFNEA</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification</title><source>arXiv.org</source><creator>Keleg, Amr ; Magdy, Walid</creator><creatorcontrib>Keleg, Amr ; Magdy, Walid</creatorcontrib><description>Automatic Arabic Dialect Identification (ADI) of text has gained great
popularity since it was introduced in the early 2010s. Multiple datasets were
developed, and yearly shared tasks have been running since 2018. However, ADI
systems are reported to fail in distinguishing between the micro-dialects of
Arabic. We argue that the currently adopted framing of the ADI task as a
single-label classification problem is one of the main reasons for that. We
highlight the limitation of the incompleteness of the Dialect labels and
demonstrate how it impacts the evaluation of ADI systems. A manual error
analysis for the predictions of an ADI, performed by 7 native speakers of
different Arabic dialects, revealed that $\approx$ 66% of the validated errors
are not true errors. Consequently, we propose framing ADI as a multi-label
classification task and give recommendations for designing new ADI datasets.</description><identifier>DOI: 10.48550/arxiv.2310.13661</identifier><language>eng</language><subject>Computer Science - Computation and Language</subject><creationdate>2023-10</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,776,881</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2310.13661$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2310.13661$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Keleg, Amr</creatorcontrib><creatorcontrib>Magdy, Walid</creatorcontrib><title>Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification</title><description>Automatic Arabic Dialect Identification (ADI) of text has gained great
popularity since it was introduced in the early 2010s. Multiple datasets were
developed, and yearly shared tasks have been running since 2018. However, ADI
systems are reported to fail in distinguishing between the micro-dialects of
Arabic. We argue that the currently adopted framing of the ADI task as a
single-label classification problem is one of the main reasons for that. We
highlight the limitation of the incompleteness of the Dialect labels and
demonstrate how it impacts the evaluation of ADI systems. A manual error
analysis for the predictions of an ADI, performed by 7 native speakers of
different Arabic dialects, revealed that $\approx$ 66% of the validated errors
are not true errors. Consequently, we propose framing ADI as a multi-label
classification task and give recommendations for designing new ADI datasets.</description><subject>Computer Science - Computation and Language</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNo9j8tOwzAURL1hgQofwAr_QIod39gNuyq8KkVi0S7YRTd-oCu5LnJcRP8eCKirkWZ0RjqM3UixhFXTiDvMX_S5rNVPIZXW8pK9rTOOZPkDYfS28I3zqVAgi4UOiR-T85lvbT4WSqd73tOeyjxN_BD4ltJ79FXE0UfeRZymM3rFLgLGyV__54Ltnh533UvVvz5vunVfoTayGqUFCMY10AIGaC1aL1qUsm5adEE7bWoMAUQAuUIFwohaj0IjaivROLVgt3-3s9rwkWmP-TT8Kg6zovoGScFNEA</recordid><startdate>20231020</startdate><enddate>20231020</enddate><creator>Keleg, Amr</creator><creator>Magdy, Walid</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20231020</creationdate><title>Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification</title><author>Keleg, Amr ; Magdy, Walid</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a671-b1c44f7d5494af49cace09a11259adf6d672aff40f418a3407026b06aa6c1a7d3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Computer Science - Computation and Language</topic><toplevel>online_resources</toplevel><creatorcontrib>Keleg, Amr</creatorcontrib><creatorcontrib>Magdy, Walid</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Keleg, Amr</au><au>Magdy, Walid</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification</atitle><date>2023-10-20</date><risdate>2023</risdate><abstract>Automatic Arabic Dialect Identification (ADI) of text has gained great
popularity since it was introduced in the early 2010s. Multiple datasets were
developed, and yearly shared tasks have been running since 2018. However, ADI
systems are reported to fail in distinguishing between the micro-dialects of
Arabic. We argue that the currently adopted framing of the ADI task as a
single-label classification problem is one of the main reasons for that. We
highlight the limitation of the incompleteness of the Dialect labels and
demonstrate how it impacts the evaluation of ADI systems. A manual error
analysis for the predictions of an ADI, performed by 7 native speakers of
different Arabic dialects, revealed that $\approx$ 66% of the validated errors
are not true errors. Consequently, we propose framing ADI as a multi-label
classification task and give recommendations for designing new ADI datasets.</abstract><doi>10.48550/arxiv.2310.13661</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | DOI: 10.48550/arxiv.2310.13661 |
| ispartof | |
| issn | |
| language | eng |
| recordid | cdi_arxiv_primary_2310_13661 |
| source | arXiv.org |
| subjects | Computer Science - Computation and Language |
| title | Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-01T07%3A23%3A05IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Arabic%20Dialect%20Identification%20under%20Scrutiny:%20Limitations%20of%20Single-label%20Classification&rft.au=Keleg,%20Amr&rft.date=2023-10-20&rft_id=info:doi/10.48550/arxiv.2310.13661&rft_dat=%3Carxiv_GOX%3E2310_13661%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |