Scaling the ISLE taxonomy: development of metrics for the multi-dimensional characterization of machine translation quality

Keith J. Miller, Michelle Vanni


Abstract
The DARPA MT evaluations of the early 1990s, along with subsequent work on the MT Scale, and the International Standards for Language Engineering (ISLE) MT Evaluation framework represent two of the principal efforts in Machine Translation Evaluation (MTE) over the past decade. We describe a research program that builds on both of these efforts. This paper focuses on the selection of MT output features suggested in the ISLE framework, as well as the development of metrics for the features to be used in the study. We define each metric and describe the rationale for its development. We also discuss several of the finer points of the evaluation measures that arose as a result of verification of the measures against sample output texts from three machine translation systems.
Anthology ID:
2001.mtsummit-papers.42
Volume:
Proceedings of Machine Translation Summit VIII
Month:
September 18-22
Year:
2001
Address:
Santiago de Compostela, Spain
Editor:
Bente Maegaard
Venue:
MTSummit
SIG:
Publisher:
Note:
Pages:
Language:
URL:
https://aclanthology.org/2001.mtsummit-papers.42
DOI:
Bibkey:
Cite (ACL):
Keith J. Miller and Michelle Vanni. 2001. Scaling the ISLE taxonomy: development of metrics for the multi-dimensional characterization of machine translation quality. In Proceedings of Machine Translation Summit VIII, Santiago de Compostela, Spain.
Cite (Informal):
Scaling the ISLE taxonomy: development of metrics for the multi-dimensional characterization of machine translation quality (Miller & Vanni, MTSummit 2001)
Copy Citation:
PDF:
https://aclanthology.org/2001.mtsummit-papers.42.pdf