<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ishikawa-etal-2026-caseinterviewbench">
    <titleInfo>
        <title>CaseInterviewBench: Evaluating Agentic AI’s Purpose-Reflective Autonomy to Uncover What the User Really Needs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shin-nosuke</namePart>
        <namePart type="family">Ishikawa</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Masato</namePart>
        <namePart type="family">Todo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Atsushi</namePart>
        <namePart type="family">Yoshino</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hirotsugu</namePart>
        <namePart type="family">Ohba</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ehsan</namePart>
            <namePart type="family">Kamalloo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nicolas</namePart>
            <namePart type="family">Gontier</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nikolai</namePart>
            <namePart type="family">Rozanov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Maryam</namePart>
            <namePart type="family">Hashemzadeh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Akhil</namePart>
            <namePart type="family">Arora</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-485-9</identifier>
    </relatedItem>
    <abstract>Agentic autonomy is essential for large language models (LLMs) to function as agentic AI systems, yet existing benchmarks often emphasize operational aspects, or HOW an agent behaves, such as tool use. The ability to reconsider the user’s underlying purpose, or WHY the agent acts, remains underexplored. We define this capability as “purpose-reflective autonomy” and introduce CaseInterviewBench, an interactive, interview-style benchmark inspired by consultant recruitment. The benchmark contains 104 predefined scenarios, each with public background context, an initial request, hidden facts, and ground-truth problem formulations and resolutions. The target LLM asks questions to uncover hidden facts, then identifies the actual problem and proposes a resolution, enabling us to observe how it infers, revises, and refines its understanding of the scenario’s underlying purpose in a structured setting. We evaluate hidden-fact disclosure, final problem formulation, and resolution using rule-based metrics and LLM-as-a-judge assessments. Across several frontier and open-weight models, we found substantial room for improvement. We also observed severe violations in which some models fabricated facilitator responses during the benchmark flow, potentially creating serious risks in agentic execution.</abstract>
    <identifier type="citekey">ishikawa-etal-2026-caseinterviewbench</identifier>
    <location>
        <url>https://aclanthology.org/2026.realm-1.68/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>986</start>
            <end>1002</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CaseInterviewBench: Evaluating Agentic AI’s Purpose-Reflective Autonomy to Uncover What the User Really Needs
%A Ishikawa, Shin-nosuke
%A Todo, Masato
%A Yoshino, Atsushi
%A Ohba, Hirotsugu
%Y Kamalloo, Ehsan
%Y Gontier, Nicolas
%Y Rozanov, Nikolai
%Y Hashemzadeh, Maryam
%Y Arora, Akhil
%S Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-485-9
%F ishikawa-etal-2026-caseinterviewbench
%X Agentic autonomy is essential for large language models (LLMs) to function as agentic AI systems, yet existing benchmarks often emphasize operational aspects, or HOW an agent behaves, such as tool use. The ability to reconsider the user’s underlying purpose, or WHY the agent acts, remains underexplored. We define this capability as “purpose-reflective autonomy” and introduce CaseInterviewBench, an interactive, interview-style benchmark inspired by consultant recruitment. The benchmark contains 104 predefined scenarios, each with public background context, an initial request, hidden facts, and ground-truth problem formulations and resolutions. The target LLM asks questions to uncover hidden facts, then identifies the actual problem and proposes a resolution, enabling us to observe how it infers, revises, and refines its understanding of the scenario’s underlying purpose in a structured setting. We evaluate hidden-fact disclosure, final problem formulation, and resolution using rule-based metrics and LLM-as-a-judge assessments. Across several frontier and open-weight models, we found substantial room for improvement. We also observed severe violations in which some models fabricated facilitator responses during the benchmark flow, potentially creating serious risks in agentic execution.
%U https://aclanthology.org/2026.realm-1.68/
%P 986-1002

Markdown (Informal)

[CaseInterviewBench: Evaluating Agentic AI’s Purpose-Reflective Autonomy to Uncover What the User Really Needs](https://aclanthology.org/2026.realm-1.68/) (Ishikawa et al., REALM 2026)

• CaseInterviewBench: Evaluating Agentic AI’s Purpose-Reflective Autonomy to Uncover What the User Really Needs (Ishikawa et al., REALM 2026)

ACL

• Shin-nosuke Ishikawa, Masato Todo, Atsushi Yoshino, and Hirotsugu Ohba. 2026. CaseInterviewBench: Evaluating Agentic AI’s Purpose-Reflective Autonomy to Uncover What the User Really Needs. In Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026), pages 986–1002, Budapest, Hungary. Association for Computational Linguistics.