Rohan Kumar Nayak
Author directory2026
Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
Ilia Kuznetsov | Rohan Kumar Nayak | Alla Rozovskaya | Iryna Gurevych
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ilia Kuznetsov | Rohan Kumar Nayak | Alla Rozovskaya | Iryna Gurevych
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it difficult to compare reviews across venues and time. To address this, we introduce a new framework for the comparative study of review quality and apply it to major AI and machine learning conferences. We propose a multi-dimensional schema to serve as a proxy for quantifying review quality in terms of its utility to editors and authors. The schema is coupled with LLM-based and lightweight measurements, which we validate against a novel test set of human-labeled review quality ratings. Contradicting the popular narrative, our proxy measurements reveal no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality. Code and data are available at https://github.com/UKPLab/emnlp2026-review-quality-estimation.