Can Latent Alignments Improve Autoregressive Machine Translation?

Adi Haviv, Lior Vassertail, Omer Levy


Abstract
Latent alignment objectives such as CTC and AXE significantly improve non-autoregressive machine translation models. Can they improve autoregressive models as well? We explore the possibility of training autoregressive machine translation models with latent alignment objectives, and observe that, in practice, this approach results in degenerate models. We provide a theoretical explanation for these empirical results, and prove that latent alignment objectives are incompatible with teacher forcing.
Anthology ID:
2021.naacl-main.209
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2637–2641
Language:
URL:
https://aclanthology.org/2021.naacl-main.209
DOI:
10.18653/v1/2021.naacl-main.209
Bibkey:
Cite (ACL):
Adi Haviv, Lior Vassertail, and Omer Levy. 2021. Can Latent Alignments Improve Autoregressive Machine Translation?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2637–2641, Online. Association for Computational Linguistics.
Cite (Informal):
Can Latent Alignments Improve Autoregressive Machine Translation? (Haviv et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.209.pdf
Video:
 https://aclanthology.org/2021.naacl-main.209.mp4