Business Critical Errors: A Framework for Adaptive Quality Feedback

Craig A Stewart, Madalena Gonçalves, Marianna Buchicchio, Alon Lavie


Abstract
Frameworks such as Multidimensional Quality Metrics (MQM) provide detailed feedback on translation quality and can pinpoint concrete linguistic errors. The quality of a translation is, however, also closely tied to its utility in a particular use case. Many customers have highly subjective expectations of translation quality. Features such as register, discourse style and brand consistency can be difficult to accommodate given a broadly applied translation solution. In this presentation we will introduce the concept of Business Critical Errors (BCE). Adapted from MQM, the BCE framework provides a perspective on translation quality that allows us to be reactive and adaptive to expectation whilst also maintaining consistency in our translation evaluation. We will demonstrate tooling used at Unbabel that allows us to evaluate the performance of our MT models on BCE using specialized test suites as well as the ability of our AI evaluation models to successfully capture BCE information.
Anthology ID:
2022.amta-upg.17
Volume:
Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track)
Month:
September
Year:
2022
Address:
Orlando, USA
Editors:
Janice Campbell, Stephen Larocca, Jay Marciano, Konstantin Savenkov, Alex Yanishevsky
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
231–256
Language:
URL:
https://aclanthology.org/2022.amta-upg.17
DOI:
Bibkey:
Cite (ACL):
Craig A Stewart, Madalena Gonçalves, Marianna Buchicchio, and Alon Lavie. 2022. Business Critical Errors: A Framework for Adaptive Quality Feedback. In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pages 231–256, Orlando, USA. Association for Machine Translation in the Americas.
Cite (Informal):
Business Critical Errors: A Framework for Adaptive Quality Feedback (Stewart et al., AMTA 2022)
Copy Citation:
Presentation:
 2022.amta-upg.17.Presentation.pdf