naab: A ready-to-use plug-and-play corpus for Farsi

The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce naab, the largest publicly avai...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:arXiv.org 2024-12
Hauptverfasser: Sadra Sabouri, Rahmati, Elnaz, Gooran, Soroush, Sameti, Hossein
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Sadra Sabouri
Rahmati, Elnaz
Gooran, Soroush
Sameti, Hossein
description The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word NAAB (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.
doi_str_mv 10.48550/arxiv.2208.13486
format Article
fullrecord <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2208_13486</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2708085736</sourcerecordid><originalsourceid>FETCH-LOGICAL-a956-e2bd757a39947aa7e89727f25fbe99c1054305997a0b1ff617090099d5f1a0eb3</originalsourceid><addsrcrecordid>eNotj0FLwzAYhoMgbMz9AE8LeE79kvRrEm9jOBUGXnYvX9dEOmpbk1Xsv3dunt7Lw8vzMHYvIcstIjxS_Gm-M6XAZlLntrhhc6W1FDZXasaWKR0BQBVGIeo50x1R9cTXPHqqJ3HqxZg8H9rxQ1BXi6GliR_6OIyJhz7yLcXU3LHbQG3yy_9dsP32eb95Fbv3l7fNeifIYSG8qmqDhrRzuSEy3jqjTFAYKu_cQQLmGtA5Q1DJEAppwAE4V2OQBL7SC7a63l6KyiE2nxSn8q-svJSdiYcrMcT-a_TpVB77MXZnp1IZsGDR6EL_AvOMTnI</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2708085736</pqid></control><display><type>article</type><title>naab: A ready-to-use plug-and-play corpus for Farsi</title><source>arXiv.org</source><source>Open Access: Freely Accessible Journals by multiple vendors</source><creator>Sadra Sabouri ; Rahmati, Elnaz ; Gooran, Soroush ; Sameti, Hossein</creator><creatorcontrib>Sadra Sabouri ; Rahmati, Elnaz ; Gooran, Soroush ; Sameti, Hossein</creatorcontrib><description>The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word NAAB (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2208.13486</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Computer Science - Computation and Language ; Large language models ; Natural language processing ; Performance enhancement</subject><ispartof>arXiv.org, 2024-12</ispartof><rights>2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,776,780,881,27902</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2208.13486$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.22034/jaiai.2024.480062.1016$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Sadra Sabouri</creatorcontrib><creatorcontrib>Rahmati, Elnaz</creatorcontrib><creatorcontrib>Gooran, Soroush</creatorcontrib><creatorcontrib>Sameti, Hossein</creatorcontrib><title>naab: A ready-to-use plug-and-play corpus for Farsi</title><title>arXiv.org</title><description>The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word NAAB (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.</description><subject>Computer Science - Computation and Language</subject><subject>Large language models</subject><subject>Natural language processing</subject><subject>Performance enhancement</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>BENPR</sourceid><sourceid>GOX</sourceid><recordid>eNotj0FLwzAYhoMgbMz9AE8LeE79kvRrEm9jOBUGXnYvX9dEOmpbk1Xsv3dunt7Lw8vzMHYvIcstIjxS_Gm-M6XAZlLntrhhc6W1FDZXasaWKR0BQBVGIeo50x1R9cTXPHqqJ3HqxZg8H9rxQ1BXi6GliR_6OIyJhz7yLcXU3LHbQG3yy_9dsP32eb95Fbv3l7fNeifIYSG8qmqDhrRzuSEy3jqjTFAYKu_cQQLmGtA5Q1DJEAppwAE4V2OQBL7SC7a63l6KyiE2nxSn8q-svJSdiYcrMcT-a_TpVB77MXZnp1IZsGDR6EL_AvOMTnI</recordid><startdate>20241223</startdate><enddate>20241223</enddate><creator>Sadra Sabouri</creator><creator>Rahmati, Elnaz</creator><creator>Gooran, Soroush</creator><creator>Sameti, Hossein</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20241223</creationdate><title>naab: A ready-to-use plug-and-play corpus for Farsi</title><author>Sadra Sabouri ; Rahmati, Elnaz ; Gooran, Soroush ; Sameti, Hossein</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a956-e2bd757a39947aa7e89727f25fbe99c1054305997a0b1ff617090099d5f1a0eb3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Computation and Language</topic><topic>Large language models</topic><topic>Natural language processing</topic><topic>Performance enhancement</topic><toplevel>online_resources</toplevel><creatorcontrib>Sadra Sabouri</creatorcontrib><creatorcontrib>Rahmati, Elnaz</creatorcontrib><creatorcontrib>Gooran, Soroush</creatorcontrib><creatorcontrib>Sameti, Hossein</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Sadra Sabouri</au><au>Rahmati, Elnaz</au><au>Gooran, Soroush</au><au>Sameti, Hossein</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>naab: A ready-to-use plug-and-play corpus for Farsi</atitle><jtitle>arXiv.org</jtitle><date>2024-12-23</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word NAAB (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2208.13486</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2024-12
issn 2331-8422
language eng
recordid cdi_arxiv_primary_2208_13486
source arXiv.org; Open Access: Freely Accessible Journals by multiple vendors
subjects Computer Science - Computation and Language
Large language models
Natural language processing
Performance enhancement
title naab: A ready-to-use plug-and-play corpus for Farsi
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-07T14%3A10%3A57IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=naab:%20A%20ready-to-use%20plug-and-play%20corpus%20for%20Farsi&rft.jtitle=arXiv.org&rft.au=Sadra%20Sabouri&rft.date=2024-12-23&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2208.13486&rft_dat=%3Cproquest_arxiv%3E2708085736%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2708085736&rft_id=info:pmid/&rfr_iscdi=true