Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling

Farhana Ferdousi Liza, Marek Grzes


Abstract
We analyse Recurrent Neural Networks (RNNs) to understand the significance of multiple LSTM layers. We argue that the Weighted Finite-state Automata (WFA) trained using a spectral learning algorithm are helpful to analyse RNNs. Our results suggest that multiple LSTM layers in RNNs help learning distributed hidden states, but have a smaller impact on the ability to learn long-term dependencies. The analysis is based on the empirical results, however relevant theory (whenever possible) was discussed to justify and support our conclusions.
Anthology ID:
W19-3903
Volume:
Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges
Month:
August
Year:
2019
Address:
Florence
Editors:
Jason Eisner, Matthias Gallé, Jeffrey Heinz, Ariadna Quattoni, Guillaume Rabusseau
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
24–33
Language:
URL:
https://aclanthology.org/W19-3903
DOI:
10.18653/v1/W19-3903
Bibkey:
Cite (ACL):
Farhana Ferdousi Liza and Marek Grzes. 2019. Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 24–33, Florence. Association for Computational Linguistics.
Cite (Informal):
Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling (Liza & Grzes, ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-3903.pdf
Data
Penn Treebank