The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control

Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, Johani Innis, Lifeng Han, Goran Nenadic


Abstract
The year 2024 marks the 10th anniversary of the Multidimensional Quality Metrics (MQM) framework for analytic translation quality evaluation. The MQM error typology has been widely used by practitioners in the translation and localization industry and has served as the basis for many derivative projects. The annual Conference on Machine Translation (WMT) shared tasks on both human and automatic translation quality evaluations used the MQM error typology. The metric stands on two pillars: error typology and the scoring model. The scoring model calculates the quality score from annotation data, detailing how to convert error type and severity counts into numeric scores to determine if the content meets specifications. Previously, only the raw scoring model had been published. This April, the MQM Council published the Linear Calibrated Scoring Model, officially presented herein, along with the Non-Linear Scoring Model, which had not been published
Anthology ID:
2024.amta-presentations.6
Volume:
Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations)
Month:
September
Year:
2024
Address:
Chicago, USA
Editors:
Marianna Martindale, Janice Campbell, Konstantin Savenkov, Shivali Goel
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
75–94
Language:
URL:
https://aclanthology.org/2024.amta-presentations.6
DOI:
Bibkey:
Cite (ACL):
Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, Johani Innis, Lifeng Han, and Goran Nenadic. 2024. The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), pages 75–94, Chicago, USA. Association for Machine Translation in the Americas.
Cite (Informal):
The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control (Lommel et al., AMTA 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.amta-presentations.6.pdf