Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold

Sebastian Ruder, Ivan Vulić, Anders Søgaard


Abstract
The prototypical NLP experiment trains a standard architecture on labeled English data and optimizes for accuracy, without accounting for other dimensions such as fairness, interpretability, or computational efficiency. We show through a manual classification of recent NLP research papers that this is indeed the case and refer to it as the square one experimental setup. We observe that NLP research often goes beyond the square one setup, e.g, focusing not only on accuracy, but also on fairness or interpretability, but typically only along a single dimension. Most work targeting multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on. Such one-dimensionality of most research means we are only exploring a fraction of the NLP research search space. We provide historical and recent examples of how the square one bias has led researchers to draw false conclusions or make unwise choices, point to promising yet unexplored directions on the research manifold, and make practical recommendations to enable more multi-dimensional research. We open-source the results of our annotations to enable further analysis.
Anthology ID:
2022.findings-acl.184
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2340–2354
Language:
URL:
https://aclanthology.org/2022.findings-acl.184
DOI:
10.18653/v1/2022.findings-acl.184
Bibkey:
Cite (ACL):
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022. Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (Ruder et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.184.pdf
Code
 google-research/url-nlp