Reliably Assessing the Quality of Post-edited Translation Based on Formalized Structured Translation Specifications

Alan K. Melby, Jason Housley, Paul J. Fields, Emily Tuioti


Abstract
Post-editing of machine translation has become more common in recent years. This has created the need for a formal method of assessing the performance of post-editors in terms of whether they are able to produce post-edited target texts that follow project specifications. This paper proposes the use of formalized structured translation specifications (FSTS) as a basis for post-editor assessment. To determine if potential evaluators are able to reliably assess the quality of post-edited translations, an experiment used texts representing the work of five fictional post-editors. Two software applications were developed to facilitate the assessment: the Ruqual Specifications Writer, which aids in establishing post-editing project specifications; and Ruqual Rubric Viewer, which provides a graphical user interface for constructing a rubric in a machine-readable format. Seventeen non-experts rated the translation quality of each simulated post-edited text. Intraclass correlation analysis showed evidence that the evaluators were highly reliable in evaluating the performance of the post-editors. Thus, we assert that using FSTS specifications applied through the Ruqual software tools provides a useful basis for evaluating the quality of post-edited texts.
Anthology ID:
2012.amta-wptp.4
Volume:
Workshop on Post-Editing Technology and Practice
Month:
October 28
Year:
2012
Address:
San Diego, California, USA
Editors:
Sharon O'Brien, Michel Simard, Lucia Specia
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
Language:
URL:
https://aclanthology.org/2012.amta-wptp.4
DOI:
Bibkey:
Cite (ACL):
Alan K. Melby, Jason Housley, Paul J. Fields, and Emily Tuioti. 2012. Reliably Assessing the Quality of Post-edited Translation Based on Formalized Structured Translation Specifications. In Workshop on Post-Editing Technology and Practice, San Diego, California, USA. Association for Machine Translation in the Americas.
Cite (Informal):
Reliably Assessing the Quality of Post-edited Translation Based on Formalized Structured Translation Specifications (Melby et al., AMTA 2012)
Copy Citation:
PDF:
https://aclanthology.org/2012.amta-wptp.4.pdf