Quality Estimation with k-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation

Tu Anh Dinh, Tobias Palzer, Jan Niehues


Abstract
Providing quality scores along with Machine Translation (MT) output, so-called reference-free Quality Estimation (QE), is crucial to inform users about the reliability of the translation. We propose a model-specific, unsupervised QE approach, termed kNN-QE, that extracts information from the MT model’s training data using k-nearest neighbors. Measuring the performance of model-specific QE is not straightforward, since they provide quality scores on their own MT output, thus cannot be evaluated using benchmark QE test sets containing human quality scores on premade MT output. Therefore, we propose an automatic evaluation method that uses quality scores from reference-based metrics as gold standard instead of human-generated ones. We are the first to conduct detailed analyses and conclude that this automatic method is sufficient, and the reference-based MetricX-23 is best for the task.
Anthology ID:
2024.eamt-1.14
Volume:
Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)
Month:
June
Year:
2024
Address:
Sheffield, UK
Editors:
Carolina Scarton, Charlotte Prescott, Chris Bayliss, Chris Oakley, Joanna Wright, Stuart Wrigley, Xingyi Song, Edward Gow-Smith, Rachel Bawden, Víctor M Sánchez-Cartagena, Patrick Cadwell, Ekaterina Lapshinova-Koltunski, Vera Cabarrão, Konstantinos Chatzitheodorou, Mary Nurminen, Diptesh Kanojia, Helena Moniz
Venue:
EAMT
SIG:
Publisher:
European Association for Machine Translation (EAMT)
Note:
Pages:
133–146
Language:
URL:
https://aclanthology.org/2024.eamt-1.14
DOI:
Bibkey:
Cite (ACL):
Tu Anh Dinh, Tobias Palzer, and Jan Niehues. 2024. Quality Estimation with k-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 133–146, Sheffield, UK. European Association for Machine Translation (EAMT).
Cite (Informal):
Quality Estimation with k-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation (Anh Dinh et al., EAMT 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eamt-1.14.pdf