In the Mood for Inference: Logic-Based Natural Language Inference with Large Language Models

Bill Noble, Rasmus Blanck, Gijs Wijnholds


Abstract
In this paper we explore challenging Natural Language Inference datasets with a logic-based approach and Large Language Models (LLMs), in order to assess the validity of this hybrid strategy. We report on an experiment which combines an LLM meta-prompting strategy, eliciting logical representations, and Prover9, a first-order logic theorem prover. In addition, we experiment with the inclusion of (logical) world knowledge. Our findings suggest that (i) broad performance is sometimes on par, (ii) formula generation is rather brittle, and (iii) world knowledge aids performance relative to data annotation. We argue that these results explicate the weaknesses of both approaches. As such, we consider this study a source of inspiration for future work in the field of neuro-symbolic reasoning.
Anthology ID:
2025.naloma-1.4
Volume:
Proceedings of the 5th Workshop on Natural Logic Meets Machine Learning (NALOMA)
Month:
August
Year:
2025
Address:
Bochum, Germany
Editors:
Lasha Abzianidze, Valeria de Paiva
Venues:
NALOMA | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
33–47
Language:
URL:
https://aclanthology.org/2025.naloma-1.4/
DOI:
Bibkey:
Cite (ACL):
Bill Noble, Rasmus Blanck, and Gijs Wijnholds. 2025. In the Mood for Inference: Logic-Based Natural Language Inference with Large Language Models. In Proceedings of the 5th Workshop on Natural Logic Meets Machine Learning (NALOMA), pages 33–47, Bochum, Germany. Association for Computational Linguistics.
Cite (Informal):
In the Mood for Inference: Logic-Based Natural Language Inference with Large Language Models (Noble et al., NALOMA 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.naloma-1.4.pdf