Categorizing Semantic Representations for Neural Machine Translation

Yongjing Yin, Yafu Li, Fandong Meng, Jie Zhou, Yue Zhang


Abstract
Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks. However, they have recently been shown to suffer limitation in compositional generalization, failing to effectively learn the translation of atoms (e.g., words) and their semantic composition (e.g., modification) from seen compounds (e.g., phrases), and thus suffering from significantly weakened translation performance on unseen compounds during inference.We address this issue by introducing categorization to the source contextualized representations. The main idea is to enhance generalization by reducing sparsity and overfitting, which is achieved by finding prototypes of token representations over the training set and integrating their embeddings into the source encoding. Experiments on a dedicated MT dataset (i.e., CoGnition) show that our method reduces compositional generalization error rates by 24% error reduction. In addition, our conceptually simple method gives consistently better results than the Transformer baseline on a range of general MT datasets.
Anthology ID:
2022.coling-1.464
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5227–5239
Language:
URL:
https://aclanthology.org/2022.coling-1.464
DOI:
Bibkey:
Cite (ACL):
Yongjing Yin, Yafu Li, Fandong Meng, Jie Zhou, and Yue Zhang. 2022. Categorizing Semantic Representations for Neural Machine Translation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5227–5239, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Categorizing Semantic Representations for Neural Machine Translation (Yin et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.464.pdf