Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, Mrinmaya Sachan


Abstract
Machine translation quality estimation (QE) predicts human judgements of a translation hypothesis without seeing the reference. State-of-the-art QE systems based on pretrained language models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even without access to the reference, our model can estimate automated metrics (ρ = 60% for BLEU, ρ = 51% for other metrics) at the sentence-level. Because automated metrics correlate with human judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ = 23%) than training for scratch (ρ = 20%).
Anthology ID:
2023.eacl-main.95
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1311–1325
Language:
URL:
https://aclanthology.org/2023.eacl-main.95
DOI:
10.18653/v1/2023.eacl-main.95
Bibkey:
Cite (ACL):
Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, and Mrinmaya Sachan. 2023. Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1311–1325, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference (Zouhar et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.95.pdf
Software:
 2023.eacl-main.95.software.zip
Video:
 https://aclanthology.org/2023.eacl-main.95.mp4