A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English

Toshiyuki Takezawa, Fumiaki Sugaya, Akio Yokoo, Seiichi Yamamoto


Abstract
ATR-MATRIX is a multi-lingual speech-to-speech translation system designed to facilitate communications between two parties of different languages engaged in a spontaneous conversation in a travel arrangement domain. In this paper, we propose a new evaluation method for speech translation systems. Our current focus is on measuring the robustness of a language translation sub-system, with quick calculation and low cost. Therefore, we calculate the difference between the translation output from transcription texts and the translation output from input speech by a dynamic programming method. We present the first trial experiment of this method applied to our Japanese-to-English speech translation system. We also provide related discussions on such points as error analysis and the relationship between the proposed method and translation quality evaluation manually done by humans.
Anthology ID:
1999.mtsummit-1.44
Volume:
Proceedings of Machine Translation Summit VII
Month:
September 13-17
Year:
1999
Address:
Singapore, Singapore
Venue:
MTSummit
SIG:
Publisher:
Note:
Pages:
299–307
Language:
URL:
https://aclanthology.org/1999.mtsummit-1.44
DOI:
Bibkey:
Cite (ACL):
Toshiyuki Takezawa, Fumiaki Sugaya, Akio Yokoo, and Seiichi Yamamoto. 1999. A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English. In Proceedings of Machine Translation Summit VII, pages 299–307, Singapore, Singapore.
Cite (Informal):
A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English (Takezawa et al., MTSummit 1999)
Copy Citation:
PDF:
https://aclanthology.org/1999.mtsummit-1.44.pdf