Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning

Shivam Mhaskar, Nirmesh Shah, Mohammadi Zaki, Ashishkumar Gudmalwar, Pankaj Wasnik, Rajiv Shah


Abstract
Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and target language texts of Machine Translation models. However, our approach aims to align the number of phonemes instead, as they are closely associated with speech duration. In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL), with a focus on optimizing the alignment of phoneme counts in the source and target language sentence pairs. To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs. Moreover, we propose a student-teacher architecture within the framework of our RL approach to maintain a trade-off between the phoneme count and translation quality.
Anthology ID:
2024.findings-naacl.250
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3966–3976
Language:
URL:
https://aclanthology.org/2024.findings-naacl.250
DOI:
Bibkey:
Cite (ACL):
Shivam Mhaskar, Nirmesh Shah, Mohammadi Zaki, Ashishkumar Gudmalwar, Pankaj Wasnik, and Rajiv Shah. 2024. Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3966–3976, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning (Mhaskar et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.250.pdf
Copyright:
 2024.findings-naacl.250.copyright.pdf