Training Hybrid Language Models by Marginalizing over Segmentations
Edouard
Grave
author
Sainbayar
Sukhbaatar
author
Piotr
Bojanowski
author
Armand
Joulin
author
2019-07
text
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Association for Computational Linguistics
Florence, Italy
conference publication
In this paper, we study the problem of hybrid language modeling, that is using models which can predict both characters and larger units such as character ngrams or words. Using such models, multiple potential segmentations usually exist for a given string, for example one using words and one using characters only. Thus, the probability of a string is the sum of the probabilities of all the possible segmentations. Here, we show how it is possible to marginalize over the segmentations efficiently, in order to compute the true probability of a sequence. We apply our technique on three datasets, comprising seven languages, showing improvements over a strong character level language model.
grave-etal-2019-training
10.18653/v1/P19-1143
https://aclanthology.org/P19-1143
2019-07
1477
1482