Unbabel’s Participation in the WMT20 Metrics Shared Task

Ricardo Rei, Craig Stewart, Ana C Farinha, Alon Lavie


Abstract
We present the contribution of the Unbabel team to the WMT 2020 Shared Task on Metrics. We intend to participate on the segmentlevel, document-level and system-level tracks on all language pairs, as well as the “QE as a Metric” track. Accordingly, we illustrate results of our models in these tracks with reference to test sets from the previous year. Our submissions build upon the recently proposed COMET framework: we train several estimator models to regress on different humangenerated quality scores and a novel ranking model trained on relative ranks obtained from Direct Assessments. We also propose a simple technique for converting segment-level predictions into a document-level score. Overall, our systems achieve strong results for all language pairs on previous test sets and in many cases set a new state-of-the-art.
Anthology ID:
2020.wmt-1.101
Volume:
Proceedings of the Fifth Conference on Machine Translation
Month:
November
Year:
2020
Address:
Online
Venues:
EMNLP | WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
911–920
Language:
URL:
https://aclanthology.org/2020.wmt-1.101
DOI:
Bibkey:
Cite (ACL):
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Unbabel’s Participation in the WMT20 Metrics Shared Task. In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics.
Cite (Informal):
Unbabel’s Participation in the WMT20 Metrics Shared Task (Rei et al., WMT 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.wmt-1.101.pdf
Optional supplementary material:
 2020.wmt-1.101.OptionalSupplementaryMaterial.pdf
Video:
 https://slideslive.com/38939604