Multi-Hypothesis Machine Translation Evaluation

Marina Fomicheva, Lucia Specia, Francisco Guzmán


Abstract
Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. One of the main challenges is the fact that multiple outputs can be equally valid. Attempts to minimise this issue include metrics that relax the matching of MT output and reference strings, and the use of multiple references. The latter has been shown to significantly improve the performance of evaluation metrics. However, collecting multiple references is expensive and in practice a single reference is generally used. In this paper, we propose an alternative approach: instead of modelling linguistic variation in human reference we exploit the MT model uncertainty to generate multiple diverse translations and use these: (i) as surrogates to reference translations; (ii) to obtain a quantification of translation variability to either complement existing metric scores or (iii) replace references altogether. We show that for a number of popular evaluation metrics our variability estimates lead to substantial improvements in correlation with human judgements of quality by up 15%.
Anthology ID:
2020.acl-main.113
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1218–1232
Language:
URL:
https://aclanthology.org/2020.acl-main.113
DOI:
10.18653/v1/2020.acl-main.113
Bibkey:
Cite (ACL):
Marina Fomicheva, Lucia Specia, and Francisco Guzmán. 2020. Multi-Hypothesis Machine Translation Evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1218–1232, Online. Association for Computational Linguistics.
Cite (Informal):
Multi-Hypothesis Machine Translation Evaluation (Fomicheva et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.113.pdf
Video:
 http://slideslive.com/38929458