An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets

Yuqiao Wen, Guoqing Luo, Lili Mou


Abstract
Open-domain dialogue systems aim to converse with humans through text, and dialogue research has heavily relied on benchmark datasets. In this work, we observe the overlapping problem in DailyDialog and OpenSubtitles, two popular open-domain dialogue benchmark datasets. Our systematic analysis then shows that such overlapping can be exploited to obtain fake state-of-the-art performance. Finally, we address this issue by cleaning these datasets and setting up a proper data processing procedure for future research.
Anthology ID:
2022.lrec-1.16
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
146–153
Language:
URL:
https://aclanthology.org/2022.lrec-1.16
DOI:
Bibkey:
Cite (ACL):
Yuqiao Wen, Guoqing Luo, and Lili Mou. 2022. An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 146–153, Marseille, France. European Language Resources Association.
Cite (Informal):
An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasets (Wen et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.16.pdf
Code
 yq-wen/overlapping-datasets
Data
ATIS