Multiple Captions Embellished Multilingual Multi-Modal Neural Machine Translation

Salam Michael Singh, Loitongbam Sanayai Meetei, Thoudam Doren Singh, Sivaji Bandyopadhyay


Abstract
Neural machine translation based on bilingual text with limited training data suffers from lexical diversity, which lowers the rare word translation accuracy and reduces the generalizability of the translation system. In this work, we utilise the multiple captions from the Multi-30K dataset to increase the lexical diversity aided with the cross-lingual transfer of information among the languages in a multilingual setup. In this multilingual and multimodal setting, the inclusion of the visual features boosts the translation quality by a significant margin. Empirical study affirms that our proposed multimodal approach achieves substantial gain in terms of the automatic score and shows robustness in handling the rare word translation in the pretext of English to/from Hindi and Telugu translation tasks.
Anthology ID:
2021.mmtlrl-1.2
Volume:
Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021)
Month:
September
Year:
2021
Address:
Online (Virtual Mode)
Editors:
Thoudam Doren Singh, Cristina España i Bonet, Sivaji Bandyopadhyay, Josef van Genabith
Venue:
MMTLRL
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
2–11
Language:
URL:
https://aclanthology.org/2021.mmtlrl-1.2
DOI:
Bibkey:
Cite (ACL):
Salam Michael Singh, Loitongbam Sanayai Meetei, Thoudam Doren Singh, and Sivaji Bandyopadhyay. 2021. Multiple Captions Embellished Multilingual Multi-Modal Neural Machine Translation. In Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021), pages 2–11, Online (Virtual Mode). INCOMA Ltd..
Cite (Informal):
Multiple Captions Embellished Multilingual Multi-Modal Neural Machine Translation (Singh et al., MMTLRL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.mmtlrl-1.2.pdf