Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications

Shuo Sun, Ahmed El-Kishky, Vishrav Chaudhary, James Cross, Lucia Specia, Francisco Guzmán


Abstract
Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent QE models have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized language models that are computationally expensive and make them infeasible for real-world applications. In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we argue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect their actual performance in real-world applications.
Anthology ID:
2021.emnlp-main.474
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5865–5875
Language:
URL:
https://aclanthology.org/2021.emnlp-main.474
DOI:
10.18653/v1/2021.emnlp-main.474
Bibkey:
Cite (ACL):
Shuo Sun, Ahmed El-Kishky, Vishrav Chaudhary, James Cross, Lucia Specia, and Francisco Guzmán. 2021. Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5865–5875, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications (Sun et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.474.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.474.mp4
Data
MLQE-PE