Predicting intelligibility from fidelity in MT evaluation

John White


Abstract
Attempts to formulate methods of automatically evaluating machine translation (MT) have generally looked at some attrinbute of translation and then tried, explicitly or implicitly, to extrapolate the measurement to cover a broader class of attributes. In particular, some studies have focused on measuring fidelity of translation, and inferring intelligibility from that, and others have taken the opposite approach. In this paper we examine the more fundamental question of whether, and to what extent, the one attribute can be predicted by the other. As a starting point we use the 1994 DARPA MT corpus, which has measures for both attributes, and perform a simple comparison of the behavior of each. Two hypotheses about a predictable inference between fidelity and intelligibility are compared with the comparative behavior across all language pairs and all documents in the corpus.
Anthology ID:
2001.mtsummit-eval.10
Volume:
Workshop on MT Evaluation
Month:
September 18-22
Year:
2001
Address:
Santiago de Compostela, Spain
Editors:
Eduard Hovy, Margaret King, Sandra Manzi, Florence Reeder
Venue:
MTSummit
SIG:
Publisher:
Note:
Pages:
Language:
URL:
https://aclanthology.org/2001.mtsummit-eval.10
DOI:
Bibkey:
Cite (ACL):
John White. 2001. Predicting intelligibility from fidelity in MT evaluation. In Workshop on MT Evaluation, Santiago de Compostela, Spain.
Cite (Informal):
Predicting intelligibility from fidelity in MT evaluation (White, MTSummit 2001)
Copy Citation:
PDF:
https://aclanthology.org/2001.mtsummit-eval.10.pdf