Neuro-Symbolic Reinforcement Learning with First-Order Logic

Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agravante, Michiaki Tatsubori, Asim Munawar, Alexander Gray


Abstract
Deep reinforcement learning (RL) methods often require many trials before convergence, and no direct interpretability of trained policies is provided. In order to achieve fast convergence and interpretability for the policy in RL, we propose a novel RL method for text-based games with a recent neuro-symbolic framework called Logical Neural Network, which can learn symbolic and interpretable rules in their differentiable network. The method is first to extract first-order logical facts from text observation and external word meaning network (ConceptNet), then train a policy in the network with directly interpretable logical operators. Our experimental results show RL training with the proposed method converges significantly faster than other state-of-the-art neuro-symbolic methods in a TextWorld benchmark.
Anthology ID:
2021.emnlp-main.283
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3505–3511
Language:
URL:
https://aclanthology.org/2021.emnlp-main.283
DOI:
10.18653/v1/2021.emnlp-main.283
Bibkey:
Cite (ACL):
Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agravante, Michiaki Tatsubori, Asim Munawar, and Alexander Gray. 2021. Neuro-Symbolic Reinforcement Learning with First-Order Logic. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3505–3511, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Neuro-Symbolic Reinforcement Learning with First-Order Logic (Kimura et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.283.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.283.mp4