Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning

Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, James Glass


Abstract
How can we perform computations over natural language representations to solve tasks that require symbolic and numeric reasoning? We propose natural language embedded programs (NLEP) as a unifying framework for addressing math/symbolic reasoning, natural language understanding, and instruction following tasks. Our approach prompts a language model to generate full Python programs that define functions over data structures which contain natural language representations of structured knowledge. A Python interpreter then executes the generated code and prints the output. Despite using a task-general prompt, we find that this approach can improve upon strong baselines across a range of different tasks including math and symbolic reasoning, text classification, question answering, and instruction following. We found that the generated programs are interpretable since they outline the exact reasoning process followed by the program interpreter.
Anthology ID:
2024.findings-naacl.259
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4131–4155
Language:
URL:
https://aclanthology.org/2024.findings-naacl.259
DOI:
Bibkey:
Cite (ACL):
Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, and James Glass. 2024. Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4131–4155, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning (Zhang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.259.pdf
Copyright:
 2024.findings-naacl.259.copyright.pdf