Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation

Xuanli He, Gholamreza Haffari, Mohammad Norouzi


Abstract
This paper introduces Dynamic Programming Encoding (DPE), a new segmentation algorithm for tokenizing sentences into subword units. We view the subword segmentation of output sentences as a latent variable that should be marginalized out for learning and inference. A mixed character-subword transformer is proposed, which enables exact log marginal likelihood estimation and exact MAP inference to find target segmentations with maximum posterior probability. DPE uses a lightweight mixed character-subword transformer as a means of pre-processing parallel data to segment output sentences using dynamic programming. Empirical results on machine translation suggest that DPE is effective for segmenting output sentences and can be combined with BPE dropout for stochastic segmentation of source sentences. DPE achieves an average improvement of 0.9 BLEU over BPE (Sennrich et al., 2016) and an average improvement of 0.55 BLEU over BPE dropout (Provilkov et al., 2019) on several WMT datasets including English <=> (German, Romanian, Estonian, Finnish, Hungarian).
Anthology ID:
2020.acl-main.275
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3042–3051
Language:
URL:
https://aclanthology.org/2020.acl-main.275
DOI:
10.18653/v1/2020.acl-main.275
Bibkey:
Cite (ACL):
Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2020. Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3042–3051, Online. Association for Computational Linguistics.
Cite (Informal):
Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation (He et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.275.pdf
Video:
 http://slideslive.com/38929282
Code
 xlhex/dpe