A test suite for evaluation of English-to-Korean machine translation systems

Sungryong Koh, Jinee Maeng, Ji-Young Lee, Young-Sook Chae, Key-Sun Choi


Abstract
This paper describes KORTERM’s test suite and their practicability. The test-sets have been being constructed on the basis of fine-grained classification of linguistic phenomena to evaluate the technical status of English-to-Korean MT systems systematically. They consist of about 5000 test-sets and are growing. Each test-set contains an English sentence, a model Korean translation, a linguistic phenomenon category, and a yes/no question about the linguistic phenomenon. Two commercial systems were evaluated with a yes/no test of prepared questions. Total accuracy rates of the two systems were different (50% vs. 66%). In addition, a comprehension test was carried out. We found that one system was more comprehensible than the other system. These results seem to show that our test suite is practicable.
Anthology ID:
2001.mtsummit-papers.35
Volume:
Proceedings of Machine Translation Summit VIII
Month:
September 18-22
Year:
2001
Address:
Santiago de Compostela, Spain
Venue:
MTSummit
SIG:
Publisher:
Note:
Pages:
Language:
URL:
https://aclanthology.org/2001.mtsummit-papers.35
DOI:
Bibkey:
Cite (ACL):
Sungryong Koh, Jinee Maeng, Ji-Young Lee, Young-Sook Chae, and Key-Sun Choi. 2001. A test suite for evaluation of English-to-Korean machine translation systems. In Proceedings of Machine Translation Summit VIII, Santiago de Compostela, Spain.
Cite (Informal):
A test suite for evaluation of English-to-Korean machine translation systems (Koh et al., MTSummit 2001)
Copy Citation:
PDF:
https://aclanthology.org/2001.mtsummit-papers.35.pdf