Evaluating Step-by-Step Reasoning through Symbolic Verification

YiFan Zhang, Hanlin Zhang, Li Li, Eric Xing


Abstract
Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually presumed to be more approachable for symbolic programming. To understand the mechanism of reasoning of LMs, we curate synthetic datasets containing equivalent (natural, symbolic) data pairs, where symbolic examples contain first-order logic rules and predicates from non-parametric knowledge bases (KBs), supporting automated verification of intermediate reasoning results. Then we revisit neuro-symbolic approaches and propose to learn from demonstrations containing logic rules and corresponding examples to iteratively reason over KBs, recovering Prolog’s backward chaining algorithm and supporting automated verification of LMs’ outputs. Comprehensive experiments are included to systematically compare LMLP with CoT in deductive reasoning settings, showing that LMLP enjoys more than 25% higher accuracy than CoT on length generalization benchmarks even with smaller model sizes.
Anthology ID:
2024.findings-naacl.188
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2984–3002
Language:
URL:
https://aclanthology.org/2024.findings-naacl.188
DOI:
Bibkey:
Cite (ACL):
YiFan Zhang, Hanlin Zhang, Li Li, and Eric Xing. 2024. Evaluating Step-by-Step Reasoning through Symbolic Verification. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2984–3002, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Evaluating Step-by-Step Reasoning through Symbolic Verification (Zhang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.188.pdf
Copyright:
 2024.findings-naacl.188.copyright.pdf