Enhancing Translation Quality by Leveraging Semantic Diversity in Multimodal Machine Translation

Ali Hatami, Mihael Arcan, Paul Buitelaar


Abstract
Despite advancements in neural machine translation, word sense disambiguation remains challenging, particularly with limited textual context. Multimodal Machine Translation enhances text-only models by integrating visual information, but its impact varies across translations. This study focuses on ambiguous sentences to investigate the effectiveness of utilizing visual information. By prioritizing these sentences, which benefit from visual cues, we aim to enhance hybrid multimodal and text-only translation approaches. We utilize Latent Semantic Analysis and Sentence-BERT to extract context vectors from the British National Corpus, enabling the assessment of semantic diversity. Our approach enhances translation quality for English-German and English-French on Multi30k, assessed through metrics including BLEU, chrF2, and TER.
Anthology ID:
2024.amta-research.14
Volume:
Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)
Month:
September
Year:
2024
Address:
Chicago, USA
Editors:
Rebecca Knowles, Akiko Eriguchi, Shivali Goel
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
154–166
Language:
URL:
https://aclanthology.org/2024.amta-research.14
DOI:
Bibkey:
Cite (ACL):
Ali Hatami, Mihael Arcan, and Paul Buitelaar. 2024. Enhancing Translation Quality by Leveraging Semantic Diversity in Multimodal Machine Translation. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 154–166, Chicago, USA. Association for Machine Translation in the Americas.
Cite (Informal):
Enhancing Translation Quality by Leveraging Semantic Diversity in Multimodal Machine Translation (Hatami et al., AMTA 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.amta-research.14.pdf