Lingua: Addressing Scenarios for Live Interpretation and Automatic Dubbing

Nathan Anderson, Caleb Wilson, Stephen D. Richardson


Abstract
Lingua is an application developed for the Church of Jesus Christ of Latter-day Saints that performs both real-time interpretation of live speeches and automatic video dubbing (AVD). Like other AVD systems, it can perform synchronized automatic dubbing, given video files and optionally, corresponding text files using a traditional ASR–MT–TTS pipeline. Lingua’s unique contribution is that it can also operate in real-time with a slight delay of a few seconds to interpret live speeches. If no source-language script is provided, the translations are exactly as recognized by ASR and translated by MT. If a script is provided, Lingua matches the recognized ASR segments with script segments and passes the latter to MT for translation and subsequent TTS. If a human translation is also provided, it is passed directly to TTS. Lingua switches between these modes dynamically, enabling translation of off-script comments and different levels of quality for multiple languages. (see extended abstract)
Anthology ID:
2022.amta-upg.14
Volume:
Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track)
Month:
September
Year:
2022
Address:
Orlando, USA
Editors:
Janice Campbell, Stephen Larocca, Jay Marciano, Konstantin Savenkov, Alex Yanishevsky
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
202–209
Language:
URL:
https://aclanthology.org/2022.amta-upg.14
DOI:
Bibkey:
Cite (ACL):
Nathan Anderson, Caleb Wilson, and Stephen D. Richardson. 2022. Lingua: Addressing Scenarios for Live Interpretation and Automatic Dubbing. In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pages 202–209, Orlando, USA. Association for Machine Translation in the Americas.
Cite (Informal):
Lingua: Addressing Scenarios for Live Interpretation and Automatic Dubbing (Anderson et al., AMTA 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.amta-upg.14.pdf