Perception vs. reality: measuring machine translation post-editing productivity

Federico Gaspari, Antonio Toral, Sudip Kumar Naskar, Declan Groves, Andy Way


Abstract
This paper presents a study of user-perceived vs real machine translation (MT) post-editing effort and productivity gains, focusing on two bidirectional language pairs: English—German and English—Dutch. Twenty experienced media professionals post-edited statistical MT output and also manually translated comparative texts within a production environment. The paper compares the actual post-editing time against the users’ perception of the effort and time required to post-edit the MT output to achieve publishable quality, thus measuring real (vs perceived) productivity gains. Although for all the language pairs users perceived MT post-editing to be slower, in fact it proved to be a faster option than manual translation for two translation directions out of four, i.e. for Dutch to English, and (marginally) for English to German. For further objective scrutiny, the paper also checks the correlation of three state-of-the-art automatic MT evaluation metrics (BLEU, METEOR and TER) with the actual post-editing time.
Anthology ID:
2014.amta-wptp.5
Volume:
Proceedings of the 11th Conference of the Association for Machine Translation in the Americas
Month:
October 22-26
Year:
2014
Address:
Vancouver, Canada
Editors:
Sharon O'Brien, Michel Simard, Lucia Specia
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
60–72
Language:
URL:
https://aclanthology.org/2014.amta-wptp.5
DOI:
Bibkey:
Cite (ACL):
Federico Gaspari, Antonio Toral, Sudip Kumar Naskar, Declan Groves, and Andy Way. 2014. Perception vs. reality: measuring machine translation post-editing productivity. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, pages 60–72, Vancouver, Canada. Association for Machine Translation in the Americas.
Cite (Informal):
Perception vs. reality: measuring machine translation post-editing productivity (Gaspari et al., AMTA 2014)
Copy Citation:
PDF:
https://aclanthology.org/2014.amta-wptp.5.pdf