Findings of the WMT26 Shared Task on Automated Translation Quality Evaluation Systems: Compact Open Models are Competitive Quality Evaluators

Alon Lavie, Greg Hanneman, Stefano Perrella, Shuoyang Ding, Eleftherios Avramidis, Lorenzo Proietti, Chi-Kiu Lo, Ammon Shurtz, Chrysoula Zerva, Archchana Sindhujan, Vilem Zouhar, Diptesh Kanojia, Frederic Blain, Brian Thompson, Giorgos Filandrianos, Orfeas Menis Mastromichalakis, Tom Kocmi, Pranav Gupta

Abstract
We present the findings of the WMT26 Shared Task on Automated Translation Quality Evaluation Systems, continuing last year’s unification of the earlier separate WMT Metrics and Quality Estimation shared tasks. This year we evaluated three complementary views of segment-level translation quality on a common test set: fine-grained error-span detection, continuous quality-score prediction, and a new task on identifying error-free translations. Submissions to upstream tasks were also converted automatically to downstream predictions. The evaluation covered 21 translation directions using human cESA judgments from the WMT26 General Machine Translation task, with optional reference translations generated as either native, post-edited, or pseudo-references, depending on the translation direction. Official evaluation data was complemented by five submitted challenge sets. Across all three primary tasks, unsupervised LLM-as-a-judge approaches outperformed all traditional and supervised metrics, while open-weight models such as Gemma 4 were found to be largely competitive with proprietary frontier models. Reference-free LLM judges are highly competitive, and the benefit of adding a reference largely depends on its provenance and on the evaluator model family.
Anthology ID:
2026.wmt-main.49
Volume:
Proceedings of the Eleventh Conference on Machine Translation
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Barry Haddow, Tom Kocmi, Philipp Koehn, Christof Monz
Venues:
WMT | WS
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
924–1023
Language:
URL:
https://aclanthology.org/2026.wmt-main.49/
DOI:
Bibkey:
Cite (ACL):
Alon Lavie, Greg Hanneman, Stefano Perrella, Shuoyang Ding, Eleftherios Avramidis, Lorenzo Proietti, Chi-Kiu Lo, Ammon Shurtz, Chrysoula Zerva, Archchana Sindhujan, Vilem Zouhar, Diptesh Kanojia, Frederic Blain, Brian Thompson, Giorgos Filandrianos, Orfeas Menis Mastromichalakis, Tom Kocmi, and Pranav Gupta. 2026. Findings of the WMT26 Shared Task on Automated Translation Quality Evaluation Systems: Compact Open Models are Competitive Quality Evaluators. In Proceedings of the Eleventh Conference on Machine Translation, pages 924–1023, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Findings of the WMT26 Shared Task on Automated Translation Quality Evaluation Systems: Compact Open Models are Competitive Quality Evaluators (Lavie et al., WMT 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.wmt-main.49.pdf
PDF Cite Search Fix data