Interpolated Spectral NGram Language Models

Ariadna Quattoni, Xavier Carreras


Abstract
Spectral models for learning weighted non-deterministic automata have nice theoretical and algorithmic properties. Despite this, it has been challenging to obtain competitive results in language modeling tasks, for two main reasons. First, in order to capture long-range dependencies of the data, the method must use statistics from long substrings, which results in very large matrices that are difficult to decompose. The second is that the loss function behind spectral learning, based on moment matching, differs from the probabilistic metrics used to evaluate language models. In this work we employ a technique for scaling up spectral learning, and use interpolated predictions that are optimized to maximize perplexity. Our experiments in character-based language modeling show that our method matches the performance of state-of-the-art ngram models, while being very fast to train.
Anthology ID:
P19-1594
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5926–5930
Language:
URL:
https://aclanthology.org/P19-1594
DOI:
10.18653/v1/P19-1594
Bibkey:
Cite (ACL):
Ariadna Quattoni and Xavier Carreras. 2019. Interpolated Spectral NGram Language Models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5926–5930, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Interpolated Spectral NGram Language Models (Quattoni & Carreras, ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1594.pdf
Video:
 https://vimeo.com/385244604