An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets

Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Wen, Yuqiao, Luo, Guoqing, Mou, Lili
Format:	Artikel
Sprache:	eng
Schlagworte:	Computer Science - Artificial Intelligence Computer Science - Computation and Language
Online-Zugang:	Volltext bestellen
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

container_end_page
container_issue
container_start_page
container_title
container_volume
creator	Wen, Yuqiao Luo, Guoqing Mou, Lili
description	Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.
doi_str_mv	10.48550/arxiv.2201.06219
format	Article
fullrecord	<record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2201_06219</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2201_06219</sourcerecordid><originalsourceid>FETCH-LOGICAL-a679-31b063267da036c255e535cd06f461879f83ca7161552122ca961d066987fa5a3</originalsourceid><addsrcrecordid>eNotz71OwzAYhWEvDKhwAUz4BhL8U9vxWDXlR6oIEt2jr45dLDm25aQVvXugdDrDKx3pQeiBknrZCEGeoHz7U80YoTWRjOpb9L6KeDNmX7yBgD_n43DGKeL5y-LuZEuAnH084I-S9sGOODncZRurNo3gI249hHQ4WtzCDJOdpzt04yBM9v66C7R73uzWr9W2e3lbr7YVSKUrTvdEcibVAIRLw4SwggszEOmWkjZKu4YbUFRSIRhlzICW9LdK3SgHAvgCPf7fXkB9Ln6Ecu7_YP0Fxn8Av5JG2g</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets</title><source>arXiv.org</source><creator>Wen, Yuqiao ; Luo, Guoqing ; Mou, Lili</creator><creatorcontrib>Wen, Yuqiao ; Luo, Guoqing ; Mou, Lili</creatorcontrib><description>Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.</description><identifier>DOI: 10.48550/arxiv.2201.06219</identifier><language>eng</language><subject>Computer Science - Artificial Intelligence ; Computer Science - Computation and Language</subject><creationdate>2022-01</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,776,881</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2201.06219$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2201.06219$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Wen, Yuqiao</creatorcontrib><creatorcontrib>Luo, Guoqing</creatorcontrib><creatorcontrib>Mou, Lili</creatorcontrib><title>An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets</title><description>Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.</description><subject>Computer Science - Artificial Intelligence</subject><subject>Computer Science - Computation and Language</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotz71OwzAYhWEvDKhwAUz4BhL8U9vxWDXlR6oIEt2jr45dLDm25aQVvXugdDrDKx3pQeiBknrZCEGeoHz7U80YoTWRjOpb9L6KeDNmX7yBgD_n43DGKeL5y-LuZEuAnH084I-S9sGOODncZRurNo3gI249hHQ4WtzCDJOdpzt04yBM9v66C7R73uzWr9W2e3lbr7YVSKUrTvdEcibVAIRLw4SwggszEOmWkjZKu4YbUFRSIRhlzICW9LdK3SgHAvgCPf7fXkB9Ln6Ecu7_YP0Fxn8Av5JG2g</recordid><startdate>20220117</startdate><enddate>20220117</enddate><creator>Wen, Yuqiao</creator><creator>Luo, Guoqing</creator><creator>Mou, Lili</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20220117</creationdate><title>An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets</title><author>Wen, Yuqiao ; Luo, Guoqing ; Mou, Lili</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a679-31b063267da036c255e535cd06f461879f83ca7161552122ca961d066987fa5a3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Computer Science - Artificial Intelligence</topic><topic>Computer Science - Computation and Language</topic><toplevel>online_resources</toplevel><creatorcontrib>Wen, Yuqiao</creatorcontrib><creatorcontrib>Luo, Guoqing</creatorcontrib><creatorcontrib>Mou, Lili</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Wen, Yuqiao</au><au>Luo, Guoqing</au><au>Mou, Lili</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets</atitle><date>2022-01-17</date><risdate>2022</risdate><abstract>Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.</abstract><doi>10.48550/arxiv.2201.06219</doi><oa>free_for_read</oa></addata></record>
fulltext	fulltext_linktorsrc
identifier	DOI: 10.48550/arxiv.2201.06219
ispartof
issn
language	eng
recordid	cdi_arxiv_primary_2201_06219
source	arXiv.org
subjects	Computer Science - Artificial Intelligence Computer Science - Computation and Language
title	An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets
url	https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-22T01%3A36%3A09IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=An%20Empirical%20Study%20on%20the%20Overlapping%20Problem%20of%20Open-Domain%20Dialogue%20Datasets&rft.au=Wen,%20Yuqiao&rft.date=2022-01-17&rft_id=info:doi/10.48550/arxiv.2201.06219&rft_dat=%3Carxiv_GOX%3E2201_06219%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true