Atsushi Yoshino

Author directory

2026

Agentic autonomy is essential for large language models (LLMs) to function as agentic AI systems, yet existing benchmarks often emphasize operational aspects, or HOW an agent behaves, such as tool use. The ability to reconsider the user’s underlying purpose, or WHY the agent acts, remains underexplored. We define this capability as "purpose-reflective autonomy" and introduce CaseInterviewBench, an interactive, interview-style benchmark inspired by consultant recruitment. The benchmark contains 104 predefined scenarios, each with public background context, an initial request, hidden facts, and ground-truth problem formulations and resolutions. The target LLM asks questions to uncover hidden facts, then identifies the actual problem and proposes a resolution, enabling us to observe how it infers, revises, and refines its understanding of the scenario’s underlying purpose in a structured setting. We evaluate hidden-fact disclosure, final problem formulation, and resolution using rule-based metrics and LLM-as-a-judge assessments. Across several frontier and open-weight models, we found substantial room for improvement. We also observed severe violations in which some models fabricated facilitator responses during the benchmark flow, potentially creating serious risks in agentic execution.
Logical consistency in the outputs of large language models (LLMs) is essential for extending their applications beyond single-task assistants, including their use in AI agent systems. Although logical reasoning capabilities are often evaluated through tasks such as mathematics and coding, where it is feasible to assess the correctness of outputs in a well-defined manner, real-world problem solving requires inductive and abductive reasoning, with an emphasis on the ability to formulate and verify hypotheses. When evaluating the hypothesis formulation and verification capabilities of LLMs, the major challenge lies in preparing a variety of tasks with appropriate difficulty, avoiding those that are either too easy or unsolvable. We introduce an automated framework called HYpothesis-Driven Reasoning Assessment (HYDRA), focusing on hypothesis formulation and verification for real-world problem solving with logical reasoning. In HYDRA, all components, including situation generation, hypothesis, and evaluation, are fully performed by LLMs. HYDRA’s systematically constructed generation and evaluation strategies enable assessment using realistic, non-trivial situations that offer both diversity and clear criteria. We evaluated six state-of-the-art LLMs using HYDRA and found that their success rates ranged from 74.3% to 90.3%, indicating that even the best-performing model among those evaluated produced rubric-invalid hypothesis–verification pairs in nearly 10% of cases.

2025

The human-level performance of Large Language Models (LLMs) across various tasks has raised expectations for the potential of artificial intelligence (AI) to possess emotions someday. To explore the capability of current LLMs to express emotions in their outputs, we conducted an experiment using several LLMs (OpenAI GPT, Google Gemini, Meta Llama3, and Cohere Command R+) to role-play as agents answering questions with specified emotional states. We defined the emotional states using Russell’s Circumplex model, a well-established framework that characterizes emotions along the sleepy-activated (arousal) and pleasure-displeasure (valence) axes. We chose this model for its simplicity, utilizing two continuous parameters, which allows for better controllability in applications involving continuous changes in emotional states. The responses generated were evaluated using a sentiment analysis model, independent of the LLMs, trained on the GoEmotions dataset. The evaluation showed that the emotional states of the generated answers were consistent with the specifications, demonstrating the LLMs’ capability for emotional expression. This indicates the potential for LLM-based