Targeting the Benchmark: On Methodology in Current Natural Language Processing Research

David Schlangen


Abstract
It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.
Anthology ID:
2021.acl-short.85
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
670–674
Language:
URL:
https://aclanthology.org/2021.acl-short.85
DOI:
10.18653/v1/2021.acl-short.85
Bibkey:
Cite (ACL):
David Schlangen. 2021. Targeting the Benchmark: On Methodology in Current Natural Language Processing Research. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 670–674, Online. Association for Computational Linguistics.
Cite (Informal):
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (Schlangen, ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-short.85.pdf
Video:
 https://aclanthology.org/2021.acl-short.85.mp4
Data
Visual Question Answering