Raison d’être of the benchmark dataset: A Survey of Current Practices of Benchmark Dataset Sharing Platforms

Jaihyun Park, Sullam Jeoung


Abstract
This paper critically examines the current practices of benchmark dataset sharing in NLP and suggests a better way to inform reusers of the benchmark dataset. As the dataset sharing platform plays a key role not only in distributing the dataset but also in informing the potential reusers about the dataset, we believe data-sharing platforms should provide a comprehensive context of the datasets. We survey four benchmark dataset sharing platforms: HuggingFace, PaperswithCode, Tensorflow, and Pytorch to diagnose the current practices of how the dataset is shared which metadata is shared and omitted. To be specific, drawing on the concept of data curation which considers the future reuse when the data is made public, we advance the direction that benchmark dataset sharing platforms should take into consideration. We identify that four benchmark platforms have different practices of using metadata and there is a lack of consensus on what social impact metadata is. We believe the problem of missing a discussion around social impact in the dataset sharing platforms has to do with the failed agreement on who should be in charge. We propose that the benchmark dataset should develop social impact metadata and data curator should take a role in managing the social impact metadata.
Anthology ID:
2022.nlppower-1.1
Volume:
Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Tatiana Shavrina, Vladislav Mikhailov, Valentin Malykh, Ekaterina Artemova, Oleg Serikov, Vitaly Protasov
Venue:
nlppower
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–10
Language:
URL:
https://aclanthology.org/2022.nlppower-1.1
DOI:
10.18653/v1/2022.nlppower-1.1
Bibkey:
Cite (ACL):
Jaihyun Park and Sullam Jeoung. 2022. Raison d’être of the benchmark dataset: A Survey of Current Practices of Benchmark Dataset Sharing Platforms. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 1–10, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Raison d’être of the benchmark dataset: A Survey of Current Practices of Benchmark Dataset Sharing Platforms (Park & Jeoung, nlppower 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.nlppower-1.1.pdf
Video:
 https://aclanthology.org/2022.nlppower-1.1.mp4
Data
GLUESQuAD