Revisiting Simple Neural Probabilistic Language Models

Simeng Sun, Mohit Iyyer


Abstract
Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements. In this paper, we revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003), which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level language model benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM’s local concatenation layer, which results in small but consistent perplexity decreases across three word-level language modeling datasets.
Anthology ID:
2021.naacl-main.407
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5181–5188
Language:
URL:
https://aclanthology.org/2021.naacl-main.407
DOI:
10.18653/v1/2021.naacl-main.407
Bibkey:
Cite (ACL):
Simeng Sun and Mohit Iyyer. 2021. Revisiting Simple Neural Probabilistic Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5181–5188, Online. Association for Computational Linguistics.
Cite (Informal):
Revisiting Simple Neural Probabilistic Language Models (Sun & Iyyer, NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.407.pdf
Video:
 https://aclanthology.org/2021.naacl-main.407.mp4
Code
 SimengSun/revisit-nplm
Data
LAMBADAWikiText-103WikiText-2