On the Practical Computational Power of Finite Precision RNNs for Language Recognition
Gail
Weiss
author
Yoav
Goldberg
author
Eran
Yahav
author
2018-07
text
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Iryna
Gurevych
editor
Yusuke
Miyao
editor
Association for Computational Linguistics
Melbourne, Australia
conference publication
While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN variants have different computational power. In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU. This is achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We show empirically that the LSTM does indeed learn to effectively use the counting mechanism.
weiss-etal-2018-practical
10.18653/v1/P18-2117
https://aclanthology.org/P18-2117
2018-07
740
745