Dataset Geography: Mapping Language Data to Language Users

Fahim Faisal, Yinkai Wang, Antonios Anastasopoulos


Abstract
As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions.
Anthology ID:
2022.acl-long.239
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3381–3411
Language:
URL:
https://aclanthology.org/2022.acl-long.239
DOI:
10.18653/v1/2022.acl-long.239
Bibkey:
Cite (ACL):
Fahim Faisal, Yinkai Wang, and Antonios Anastasopoulos. 2022. Dataset Geography: Mapping Language Data to Language Users. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3381–3411, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Dataset Geography: Mapping Language Data to Language Users (Faisal et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.239.pdf
Video:
 https://aclanthology.org/2022.acl-long.239.mp4
Data
MLQAMasakhaNERNatural QuestionsSQuADTyDiQA