Measuring the Effects of Human and Machine Translation on Website Engagement

Geza Kovacs, John DeNero


Abstract
With the internet growing increasingly multilingual, it is important to consider translating websites. However, professional translators are much more expensive than machines, and machine translation quality is continually increasing, so we must justify the cost of professional translation by measuring the effects of translation on website engagement, and how users interact with translations. This paper presents an in-the-wild study run on 2 websites fully translated into 15 and 11 languages respectively, where visitors with non-English preferred languages were randomized into being shown text translated by a professional translator, machine translated text, or untranslated English text. We find that both human and machine translations improve engagement, users rarely switch the page language manually, and that in-browser machine translation is often used when English is shown, particularly by users from countries with low English proficiency. We also release a dataset of interaction data collected during our studies, including 3,332,669 sessions from 190 countries across 2 websites.
Anthology ID:
2022.amta-research.23
Volume:
Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)
Month:
September
Year:
2022
Address:
Orlando, USA
Editors:
Kevin Duh, Francisco Guzmán
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
298–308
Language:
URL:
https://aclanthology.org/2022.amta-research.23
DOI:
Bibkey:
Cite (ACL):
Geza Kovacs and John DeNero. 2022. Measuring the Effects of Human and Machine Translation on Website Engagement. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 298–308, Orlando, USA. Association for Machine Translation in the Americas.
Cite (Informal):
Measuring the Effects of Human and Machine Translation on Website Engagement (Kovacs & DeNero, AMTA 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.amta-research.23.pdf