Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing

Haoming Jiang, Chen Liang, Chong Wang, Tuo Zhao


Abstract
Many multi-domain neural machine translation (NMT) models achieve knowledge transfer by enforcing one encoder to learn shared embedding across domains. However, this design lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. We first observe that words in a sentence are often related to multiple domains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are obtained by mixing their embedding in individual domains based on their domain proportions. We show this can be achieved by carefully designing multi-head dot-product attention modules for different domains, and eventually taking weighted averages of their parameters by word-level layer-wise domain proportions. Through this, we can achieve effective domain knowledge sharing and capture fine-grained domain-specific knowledge as well. Our experiments show that our proposed model outperforms existing ones in several NMT tasks.
Anthology ID:
2020.acl-main.165
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1823–1834
Language:
URL:
https://aclanthology.org/2020.acl-main.165
DOI:
10.18653/v1/2020.acl-main.165
Bibkey:
Cite (ACL):
Haoming Jiang, Chen Liang, Chong Wang, and Tuo Zhao. 2020. Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1823–1834, Online. Association for Computational Linguistics.
Cite (Informal):
Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing (Jiang et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.165.pdf
Video:
 http://slideslive.com/38928696