Explainable Multi-hop Verbal Reasoning Through Internal Monologue

Zhengzhong Liang, Steven Bethard, Mihai Surdeanu


Abstract
Many state-of-the-art (SOTA) language models have achieved high accuracy on several multi-hop reasoning problems. However, these approaches tend to not be interpretable because they do not make the intermediate reasoning steps explicit. Moreover, models trained on simpler tasks tend to fail when directly tested on more complex problems. We propose the Explainable multi-hop Verbal Reasoner (EVR) to solve these limitations by (a) decomposing multi-hop reasoning problems into several simple ones, and (b) using natural language to guide the intermediate reasoning hops. We implement EVR by extending the classic reasoning paradigm General Problem Solver (GPS) with a SOTA generative language model to generate subgoals and perform inference in natural language at each reasoning step. Evaluation of EVR on the RuleTaker synthetic question answering (QA) dataset shows that EVR achieves SOTA performance while being able to generate all reasoning steps in natural language. Furthermore, EVR generalizes better than other strong methods when trained on simpler tasks or less training data (up to 35.7% and 7.7% absolute improvement respectively).
Anthology ID:
2021.naacl-main.97
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1225–1250
Language:
URL:
https://aclanthology.org/2021.naacl-main.97
DOI:
10.18653/v1/2021.naacl-main.97
Bibkey:
Cite (ACL):
Zhengzhong Liang, Steven Bethard, and Mihai Surdeanu. 2021. Explainable Multi-hop Verbal Reasoning Through Internal Monologue. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1225–1250, Online. Association for Computational Linguistics.
Cite (Informal):
Explainable Multi-hop Verbal Reasoning Through Internal Monologue (Liang et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.97.pdf
Video:
 https://aclanthology.org/2021.naacl-main.97.mp4