What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Terne Thorn Jakobsen, Anna Rogers


Abstract
Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.
Anthology ID:
2022.naacl-main.354
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4810–4823
Language:
URL:
https://aclanthology.org/2022.naacl-main.354
DOI:
10.18653/v1/2022.naacl-main.354
Bibkey:
Cite (ACL):
Terne Thorn Jakobsen and Anna Rogers. 2022. What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4810–4823, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review (Thorn Jakobsen & Rogers, NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.354.pdf
Code
 terne/paper-reviewer-matching-surveys