ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream app...
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext bestellen |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | |
| container_volume | |
| creator | Shi, Jiatong Tian, Jinchuan Wu, Yihan Jung, Jee-weon Yip, Jia Qi Masuyama, Yoshiki Chen, William Wu, Yuning Tang, Yuxun Baali, Massa Alharhi, Dareen Zhang, Dong Deng, Ruifan Srivastava, Tejes Wu, Haibin Liu, Alexander H Raj, Bhiksha Jin, Qin Song, Ruihua Watanabe, Shinji |
| description | Neural codecs have become crucial to recent speech and audio generation
research. In addition to signal compression capabilities, discrete codecs have
also been found to enhance downstream training efficiency and compatibility
with autoregressive language models. However, as extensive downstream
applications are investigated, challenges have arisen in ensuring fair
comparisons across diverse applications. To address these issues, we present a
new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on
neural codec training and evaluation. ESPnet-Codec offers various recipes in
audio, music, and speech for training and evaluation using several widely
adopted codec models. Together with ESPnet-Codec, we present VERSA, a
standalone evaluation toolkit, which provides a comprehensive evaluation of
codec performance over 20 audio evaluation metrics. Notably, we demonstrate
that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse
applications. |
| doi_str_mv | 10.48550/arxiv.2409.15897 |
| format | Article |
| fullrecord | <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2409_15897</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2409_15897</sourcerecordid><originalsourceid>FETCH-arxiv_primary_2409_158973</originalsourceid><addsrcrecordid>eNqFzsEKgkAUheHZtIjqAVp1H0DNSknbhRhtikD3ctFrXtAZmXGk3j6S9q3O5vzwCbHe-V4QhaG_Rf3i0dsHfuztwig-zgWm2UPS4CaqovIEiep6TQ1JwyNBrpElyyegrCAdsbU4sJKgariT1djClBmolYazrVg5cLOGS2cqsp6obJZiVmNraPXbhdhc0jy5uhOm6DV3qN_FF1VMqMP_xwcNe0GZ</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech</title><source>arXiv.org</source><creator>Shi, Jiatong ; Tian, Jinchuan ; Wu, Yihan ; Jung, Jee-weon ; Yip, Jia Qi ; Masuyama, Yoshiki ; Chen, William ; Wu, Yuning ; Tang, Yuxun ; Baali, Massa ; Alharhi, Dareen ; Zhang, Dong ; Deng, Ruifan ; Srivastava, Tejes ; Wu, Haibin ; Liu, Alexander H ; Raj, Bhiksha ; Jin, Qin ; Song, Ruihua ; Watanabe, Shinji</creator><creatorcontrib>Shi, Jiatong ; Tian, Jinchuan ; Wu, Yihan ; Jung, Jee-weon ; Yip, Jia Qi ; Masuyama, Yoshiki ; Chen, William ; Wu, Yuning ; Tang, Yuxun ; Baali, Massa ; Alharhi, Dareen ; Zhang, Dong ; Deng, Ruifan ; Srivastava, Tejes ; Wu, Haibin ; Liu, Alexander H ; Raj, Bhiksha ; Jin, Qin ; Song, Ruihua ; Watanabe, Shinji</creatorcontrib><description>Neural codecs have become crucial to recent speech and audio generation
research. In addition to signal compression capabilities, discrete codecs have
also been found to enhance downstream training efficiency and compatibility
with autoregressive language models. However, as extensive downstream
applications are investigated, challenges have arisen in ensuring fair
comparisons across diverse applications. To address these issues, we present a
new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on
neural codec training and evaluation. ESPnet-Codec offers various recipes in
audio, music, and speech for training and evaluation using several widely
adopted codec models. Together with ESPnet-Codec, we present VERSA, a
standalone evaluation toolkit, which provides a comprehensive evaluation of
codec performance over 20 audio evaluation metrics. Notably, we demonstrate
that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse
applications.</description><identifier>DOI: 10.48550/arxiv.2409.15897</identifier><language>eng</language><subject>Computer Science - Sound</subject><creationdate>2024-09</creationdate><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2409.15897$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2409.15897$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Shi, Jiatong</creatorcontrib><creatorcontrib>Tian, Jinchuan</creatorcontrib><creatorcontrib>Wu, Yihan</creatorcontrib><creatorcontrib>Jung, Jee-weon</creatorcontrib><creatorcontrib>Yip, Jia Qi</creatorcontrib><creatorcontrib>Masuyama, Yoshiki</creatorcontrib><creatorcontrib>Chen, William</creatorcontrib><creatorcontrib>Wu, Yuning</creatorcontrib><creatorcontrib>Tang, Yuxun</creatorcontrib><creatorcontrib>Baali, Massa</creatorcontrib><creatorcontrib>Alharhi, Dareen</creatorcontrib><creatorcontrib>Zhang, Dong</creatorcontrib><creatorcontrib>Deng, Ruifan</creatorcontrib><creatorcontrib>Srivastava, Tejes</creatorcontrib><creatorcontrib>Wu, Haibin</creatorcontrib><creatorcontrib>Liu, Alexander H</creatorcontrib><creatorcontrib>Raj, Bhiksha</creatorcontrib><creatorcontrib>Jin, Qin</creatorcontrib><creatorcontrib>Song, Ruihua</creatorcontrib><creatorcontrib>Watanabe, Shinji</creatorcontrib><title>ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech</title><description>Neural codecs have become crucial to recent speech and audio generation
research. In addition to signal compression capabilities, discrete codecs have
also been found to enhance downstream training efficiency and compatibility
with autoregressive language models. However, as extensive downstream
applications are investigated, challenges have arisen in ensuring fair
comparisons across diverse applications. To address these issues, we present a
new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on
neural codec training and evaluation. ESPnet-Codec offers various recipes in
audio, music, and speech for training and evaluation using several widely
adopted codec models. Together with ESPnet-Codec, we present VERSA, a
standalone evaluation toolkit, which provides a comprehensive evaluation of
codec performance over 20 audio evaluation metrics. Notably, we demonstrate
that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse
applications.</description><subject>Computer Science - Sound</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNqFzsEKgkAUheHZtIjqAVp1H0DNSknbhRhtikD3ctFrXtAZmXGk3j6S9q3O5vzwCbHe-V4QhaG_Rf3i0dsHfuztwig-zgWm2UPS4CaqovIEiep6TQ1JwyNBrpElyyegrCAdsbU4sJKgariT1djClBmolYazrVg5cLOGS2cqsp6obJZiVmNraPXbhdhc0jy5uhOm6DV3qN_FF1VMqMP_xwcNe0GZ</recordid><startdate>20240924</startdate><enddate>20240924</enddate><creator>Shi, Jiatong</creator><creator>Tian, Jinchuan</creator><creator>Wu, Yihan</creator><creator>Jung, Jee-weon</creator><creator>Yip, Jia Qi</creator><creator>Masuyama, Yoshiki</creator><creator>Chen, William</creator><creator>Wu, Yuning</creator><creator>Tang, Yuxun</creator><creator>Baali, Massa</creator><creator>Alharhi, Dareen</creator><creator>Zhang, Dong</creator><creator>Deng, Ruifan</creator><creator>Srivastava, Tejes</creator><creator>Wu, Haibin</creator><creator>Liu, Alexander H</creator><creator>Raj, Bhiksha</creator><creator>Jin, Qin</creator><creator>Song, Ruihua</creator><creator>Watanabe, Shinji</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20240924</creationdate><title>ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech</title><author>Shi, Jiatong ; Tian, Jinchuan ; Wu, Yihan ; Jung, Jee-weon ; Yip, Jia Qi ; Masuyama, Yoshiki ; Chen, William ; Wu, Yuning ; Tang, Yuxun ; Baali, Massa ; Alharhi, Dareen ; Zhang, Dong ; Deng, Ruifan ; Srivastava, Tejes ; Wu, Haibin ; Liu, Alexander H ; Raj, Bhiksha ; Jin, Qin ; Song, Ruihua ; Watanabe, Shinji</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-arxiv_primary_2409_158973</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Computer Science - Sound</topic><toplevel>online_resources</toplevel><creatorcontrib>Shi, Jiatong</creatorcontrib><creatorcontrib>Tian, Jinchuan</creatorcontrib><creatorcontrib>Wu, Yihan</creatorcontrib><creatorcontrib>Jung, Jee-weon</creatorcontrib><creatorcontrib>Yip, Jia Qi</creatorcontrib><creatorcontrib>Masuyama, Yoshiki</creatorcontrib><creatorcontrib>Chen, William</creatorcontrib><creatorcontrib>Wu, Yuning</creatorcontrib><creatorcontrib>Tang, Yuxun</creatorcontrib><creatorcontrib>Baali, Massa</creatorcontrib><creatorcontrib>Alharhi, Dareen</creatorcontrib><creatorcontrib>Zhang, Dong</creatorcontrib><creatorcontrib>Deng, Ruifan</creatorcontrib><creatorcontrib>Srivastava, Tejes</creatorcontrib><creatorcontrib>Wu, Haibin</creatorcontrib><creatorcontrib>Liu, Alexander H</creatorcontrib><creatorcontrib>Raj, Bhiksha</creatorcontrib><creatorcontrib>Jin, Qin</creatorcontrib><creatorcontrib>Song, Ruihua</creatorcontrib><creatorcontrib>Watanabe, Shinji</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Shi, Jiatong</au><au>Tian, Jinchuan</au><au>Wu, Yihan</au><au>Jung, Jee-weon</au><au>Yip, Jia Qi</au><au>Masuyama, Yoshiki</au><au>Chen, William</au><au>Wu, Yuning</au><au>Tang, Yuxun</au><au>Baali, Massa</au><au>Alharhi, Dareen</au><au>Zhang, Dong</au><au>Deng, Ruifan</au><au>Srivastava, Tejes</au><au>Wu, Haibin</au><au>Liu, Alexander H</au><au>Raj, Bhiksha</au><au>Jin, Qin</au><au>Song, Ruihua</au><au>Watanabe, Shinji</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech</atitle><date>2024-09-24</date><risdate>2024</risdate><abstract>Neural codecs have become crucial to recent speech and audio generation
research. In addition to signal compression capabilities, discrete codecs have
also been found to enhance downstream training efficiency and compatibility
with autoregressive language models. However, as extensive downstream
applications are investigated, challenges have arisen in ensuring fair
comparisons across diverse applications. To address these issues, we present a
new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on
neural codec training and evaluation. ESPnet-Codec offers various recipes in
audio, music, and speech for training and evaluation using several widely
adopted codec models. Together with ESPnet-Codec, we present VERSA, a
standalone evaluation toolkit, which provides a comprehensive evaluation of
codec performance over 20 audio evaluation metrics. Notably, we demonstrate
that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse
applications.</abstract><doi>10.48550/arxiv.2409.15897</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext_linktorsrc |
| identifier | DOI: 10.48550/arxiv.2409.15897 |
| ispartof | |
| issn | |
| language | eng |
| recordid | cdi_arxiv_primary_2409_15897 |
| source | arXiv.org |
| subjects | Computer Science - Sound |
| title | ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-11T16%3A05%3A51IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=ESPnet-Codec:%20Comprehensive%20Training%20and%20Evaluation%20of%20Neural%20Codecs%20for%20Audio,%20Music,%20and%20Speech&rft.au=Shi,%20Jiatong&rft.date=2024-09-24&rft_id=info:doi/10.48550/arxiv.2409.15897&rft_dat=%3Carxiv_GOX%3E2409_15897%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |