Masahiro Daimon
Author directory2026
HYDRA: Evaluating Logical Thinking Abilities of LLMs through Formulating Hypotheses and Verification Methods
Shin-nosuke Ishikawa | Hirotsugu Ohba | Masato Todo | Atsushi Yoshino | Masahiro Daimon
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Shin-nosuke Ishikawa | Hirotsugu Ohba | Masato Todo | Atsushi Yoshino | Masahiro Daimon
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Logical consistency in the outputs of large language models (LLMs) is essential for extending their applications beyond single-task assistants, including their use in AI agent systems. Although logical reasoning capabilities are often evaluated through tasks such as mathematics and coding, where it is feasible to assess the correctness of outputs in a well-defined manner, real-world problem solving requires inductive and abductive reasoning, with an emphasis on the ability to formulate and verify hypotheses. When evaluating the hypothesis formulation and verification capabilities of LLMs, the major challenge lies in preparing a variety of tasks with appropriate difficulty, avoiding those that are either too easy or unsolvable. We introduce an automated framework called HYpothesis-Driven Reasoning Assessment (HYDRA), focusing on hypothesis formulation and verification for real-world problem solving with logical reasoning. In HYDRA, all components, including situation generation, hypothesis, and evaluation, are fully performed by LLMs. HYDRA’s systematically constructed generation and evaluation strategies enable assessment using realistic, non-trivial situations that offer both diversity and clear criteria. We evaluated six state-of-the-art LLMs using HYDRA and found that their success rates ranged from 74.3% to 90.3%, indicating that even the best-performing model among those evaluated produced rubric-invalid hypothesis–verification pairs in nearly 10% of cases.