Evaluating the Deductive Competence of Large Language Models

S Seals, Valerie Shalin


Abstract
The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance and the human-generated language corpora that informs them.
Anthology ID:
2024.naacl-long.476
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8606–8622
Language:
URL:
https://aclanthology.org/2024.naacl-long.476
DOI:
Bibkey:
Cite (ACL):
S Seals and Valerie Shalin. 2024. Evaluating the Deductive Competence of Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8606–8622, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Evaluating the Deductive Competence of Large Language Models (Seals & Shalin, NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.476.pdf
Copyright:
 2024.naacl-long.476.copyright.pdf