Reading Comprehension of Machine Translation Output: What Makes for a Better Read?

Sheila Castilho, Ana Guerberof Arenas


Abstract
This paper reports on a pilot experiment that compares two different machine translation (MT) paradigms in reading comprehension tests. To explore a suitable methodology, we set up a pilot experiment with a group of six users (with English, Spanish and Simplified Chinese languages) using an English Language Testing System (IELTS), and an eye-tracker. The users were asked to read three texts in their native language: either the original English text (for the English speakers) or the machine-translated text (for the Spanish and Simplified Chinese speakers). The original texts were machine-translated via two MT systems: neural (NMT) and statistical (SMT). The users were also asked to rank satisfaction statements on a 3-point scale after reading each text and answering the respective comprehension questions. After all tasks were completed, a post-task retrospective interview took place to gather qualitative data. The findings suggest that the users from the target languages completed more tasks in less time with a higher level of satisfaction when using translations from the NMT system.
Anthology ID:
2018.eamt-main.8
Volume:
Proceedings of the 21st Annual Conference of the European Association for Machine Translation
Month:
May
Year:
2018
Address:
Alicante, Spain
Editors:
Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popović, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada
Venue:
EAMT
SIG:
Publisher:
Note:
Pages:
99–108
Language:
URL:
https://aclanthology.org/2018.eamt-main.8
DOI:
Bibkey:
Cite (ACL):
Sheila Castilho and Ana Guerberof Arenas. 2018. Reading Comprehension of Machine Translation Output: What Makes for a Better Read?. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, pages 99–108, Alicante, Spain.
Cite (Informal):
Reading Comprehension of Machine Translation Output: What Makes for a Better Read? (Castilho & Guerberof Arenas, EAMT 2018)
Copy Citation:
PDF:
https://aclanthology.org/2018.eamt-main.8.pdf