On Long-Tailed Phenomena in Neural Machine Translation

Vikas Raunak, Siddharth Dalmia, Vivek Gupta, Florian Metze


Abstract
State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenomena in the context of structured prediction tasks is further hindered by the added complexities of search during inference. In this work, we quantitatively characterize such long-tailed phenomena at two levels of abstraction, namely, token classification and sequence generation. We propose a new loss function, the Anti-Focal loss, to better adapt model training to the structural dependencies of conditional text generation by incorporating the inductive biases of beam search in the training process. We show the efficacy of the proposed technique on a number of Machine Translation (MT) datasets, demonstrating that it leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. We have released the code to reproduce our results.
Anthology ID:
2020.findings-emnlp.276
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3088–3095
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.276
DOI:
10.18653/v1/2020.findings-emnlp.276
Bibkey:
Cite (ACL):
Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. 2020. On Long-Tailed Phenomena in Neural Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3088–3095, Online. Association for Computational Linguistics.
Cite (Informal):
On Long-Tailed Phenomena in Neural Machine Translation (Raunak et al., Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.276.pdf
Video:
 https://slideslive.com/38940652
Code
 vyraun/long-tailed