LSTMs Compose—and Learn—Bottom-Up

Naomi Saphra, Adam Lopez


Abstract
Recent work in NLP shows that LSTM language models capture compositional structure in language data. In contrast to existing work, we consider the learning process that leads to compositional behavior. For a closer look at how an LSTM’s sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We support this measure with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hierarchical structures are discovered over the course of training: that LSTM constituent representations are learned bottom-up, relying on effective representations of their shorter children, rather than on learning the longer-range relations independently.
Anthology ID:
2020.findings-emnlp.252
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2797–2809
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.252
DOI:
10.18653/v1/2020.findings-emnlp.252
Bibkey:
Cite (ACL):
Naomi Saphra and Adam Lopez. 2020. LSTMs Compose—and Learn—Bottom-Up. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2797–2809, Online. Association for Computational Linguistics.
Cite (Informal):
LSTMs Compose—and Learn—Bottom-Up (Saphra & Lopez, Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.252.pdf