Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification

Yang Xiao, Jinlan Fu, See-Kiong Ng, Pengfei Liu


Abstract
In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first characterizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. We further, taking the text classification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promisingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset discrimination over unseen datasets. We released all datasets with features explored in this work on DataLab.
Anthology ID:
2022.naacl-main.213
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2930–2941
Language:
URL:
https://aclanthology.org/2022.naacl-main.213
DOI:
10.18653/v1/2022.naacl-main.213
Bibkey:
Cite (ACL):
Yang Xiao, Jinlan Fu, See-Kiong Ng, and Pengfei Liu. 2022. Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2930–2941, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification (Xiao et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.213.pdf
Video:
 https://aclanthology.org/2022.naacl-main.213.mp4
Data
ATISGLUEIMDb Movie ReviewsMultiNLISSTSST-2