基于图文细粒度对齐语义引导的多模态神经机器翻译方法(Based on Semantic Guidance of Fine-grained Alignment of Image-Text for Multi-modal Neural Machine Translation)

Junjie Ye (叶俊杰), Junjun Guo (郭军军), Kaiwen Tan (谭凯文), Yan Xiang (相艳), Zhengtao Yu (余正涛)


Abstract
“多模态神经机器翻译旨在利用视觉信息来提高文本翻译质量。传统多模态机器翻译将图像的全局语义信息融入到翻译模型,而忽略了图像的细粒度信息对翻译质量的影响。对此,该文提出一种基于图文细粒度对齐语义引导的多模态神经机器翻译方法,该方法首先跨模态交互图文信息,以提取图文细粒度对齐语义信息,然后以图文细粒度对齐语义信息为枢纽,采用门控机制将多模态细粒度信息对齐到文本信息上,实现图文多模态特征融合。在多模态机器翻译基准数据集Multi30K 英语→德语、英语→法语以及英语→捷克语翻译任务上的实验结果表明,论文提出方法的有效性,并且优于大多数最先进的多模态机器翻译方法。”
Anthology ID:
2022.ccl-1.26
Volume:
Proceedings of the 21st Chinese National Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Nanchang, China
Editors:
Maosong Sun (孙茂松), Yang Liu (刘洋), Wanxiang Che (车万翔), Yang Feng (冯洋), Xipeng Qiu (邱锡鹏), Gaoqi Rao (饶高琦), Yubo Chen (陈玉博)
Venue:
CCL
SIG:
Publisher:
Chinese Information Processing Society of China
Note:
Pages:
281–292
Language:
Chinese
URL:
https://aclanthology.org/2022.ccl-1.26
DOI:
Bibkey:
Cite (ACL):
Junjie Ye, Junjun Guo, Kaiwen Tan, Yan Xiang, and Zhengtao Yu. 2022. 基于图文细粒度对齐语义引导的多模态神经机器翻译方法(Based on Semantic Guidance of Fine-grained Alignment of Image-Text for Multi-modal Neural Machine Translation). In Proceedings of the 21st Chinese National Conference on Computational Linguistics, pages 281–292, Nanchang, China. Chinese Information Processing Society of China.
Cite (Informal):
基于图文细粒度对齐语义引导的多模态神经机器翻译方法(Based on Semantic Guidance of Fine-grained Alignment of Image-Text for Multi-modal Neural Machine Translation) (Ye et al., CCL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.ccl-1.26.pdf