Preregistering NLP research

Emiel van Miltenburg, Chris van der Lee, Emiel Krahmer


Abstract
Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research.
Anthology ID:
2021.naacl-main.51
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
613–623
Language:
URL:
https://aclanthology.org/2021.naacl-main.51
DOI:
10.18653/v1/2021.naacl-main.51
Award:
 Best Thematic Paper
Bibkey:
Cite (ACL):
Emiel van Miltenburg, Chris van der Lee, and Emiel Krahmer. 2021. Preregistering NLP research. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 613–623, Online. Association for Computational Linguistics.
Cite (Informal):
Preregistering NLP research (van Miltenburg et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.51.pdf
Video:
 https://aclanthology.org/2021.naacl-main.51.mp4