@inproceedings{rallabandi-etal-2026-agent,
title = "What Agent Benchmarks Don{'}t Tell You: Recoverability and Credit Assignment in Agentic Retrieval Evaluation",
author = "Rallabandi, Sai Krishna and
Dakle, Parag Pravin and
Raghavan, Preethi",
editor = "Kamalloo, Ehsan and
Gontier, Nicolas and
Rozanov, Nikolai and
Hashemzadeh, Maryam and
Arora, Akhil",
booktitle = "Proceedings of the Second Workshop for {RE}search on Agent Language Models ({REALM} 2026)",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.realm-1.47/",
pages = "687--701",
ISBN = "979-8-89176-485-9",
abstract = "Agent quality on long-horizon tasks is usually reported as a single number: the aggregate gain of a tool-using agent over a non-agentic baseline on a benchmark. We argue that this number is not a property of the agent. It is jointly determined by a property of the benchmark and by which sub-component of the agent happens to be load-bearing, and reporting it alone makes agent quality look benchmark-dependent and irreproducible. We propose a four-stage evaluation protocol that separates the two. Stage 1 characterizes the benchmark before any outcome is observed, using an LLM-free diagnostic we call recoverability: whether evidence missed on the first retrieval is still reachable from what the system has already seen, and if so whether it is reachable by reformulation or only by structural navigation. Stage 2 assigns credit with an ablation ladder {---} one-shot RAG, a choice-free fixed-iterative loop, and a full agent {---} that localizes any measured gain to iteration or to open-ended agency. Stages 3 and 4 add the validity checks and cost accounting that change verdicts in practice. Applying the protocol to about 6,000 logged agent episodes over eight corpora and three reader families, we find that the headline agent gain ranges from {\ensuremath{-}}0.21 to +0.30 (each corpus in its native metric) purely as a function of the recoverability regime, and that on reformulation-recoverable corpora the choice-free loop recovers 65{--}96{\%} of the gain normally credited to agency under the two stronger readers (and most of it under the open reader). Agency earns its cost only on the longest-horizon, navigation-recoverable corpus, where iteration saturates by round 2 and leaves a residual (+0.15 under the open reader at full N) that no round budget closes under any reader."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rallabandi-etal-2026-agent">
<titleInfo>
<title>What Agent Benchmarks Don’t Tell You: Recoverability and Credit Assignment in Agentic Retrieval Evaluation</title>
</titleInfo>
<name type="personal">
<namePart type="given">Sai</namePart>
<namePart type="given">Krishna</namePart>
<namePart type="family">Rallabandi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Parag</namePart>
<namePart type="given">Pravin</namePart>
<namePart type="family">Dakle</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Preethi</namePart>
<namePart type="family">Raghavan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)</title>
</titleInfo>
<name type="personal">
<namePart type="given">Ehsan</namePart>
<namePart type="family">Kamalloo</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nicolas</namePart>
<namePart type="family">Gontier</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nikolai</namePart>
<namePart type="family">Rozanov</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Maryam</namePart>
<namePart type="family">Hashemzadeh</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Akhil</namePart>
<namePart type="family">Arora</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-485-9</identifier>
</relatedItem>
<abstract>Agent quality on long-horizon tasks is usually reported as a single number: the aggregate gain of a tool-using agent over a non-agentic baseline on a benchmark. We argue that this number is not a property of the agent. It is jointly determined by a property of the benchmark and by which sub-component of the agent happens to be load-bearing, and reporting it alone makes agent quality look benchmark-dependent and irreproducible. We propose a four-stage evaluation protocol that separates the two. Stage 1 characterizes the benchmark before any outcome is observed, using an LLM-free diagnostic we call recoverability: whether evidence missed on the first retrieval is still reachable from what the system has already seen, and if so whether it is reachable by reformulation or only by structural navigation. Stage 2 assigns credit with an ablation ladder — one-shot RAG, a choice-free fixed-iterative loop, and a full agent — that localizes any measured gain to iteration or to open-ended agency. Stages 3 and 4 add the validity checks and cost accounting that change verdicts in practice. Applying the protocol to about 6,000 logged agent episodes over eight corpora and three reader families, we find that the headline agent gain ranges from \ensuremath-0.21 to +0.30 (each corpus in its native metric) purely as a function of the recoverability regime, and that on reformulation-recoverable corpora the choice-free loop recovers 65–96% of the gain normally credited to agency under the two stronger readers (and most of it under the open reader). Agency earns its cost only on the longest-horizon, navigation-recoverable corpus, where iteration saturates by round 2 and leaves a residual (+0.15 under the open reader at full N) that no round budget closes under any reader.</abstract>
<identifier type="citekey">rallabandi-etal-2026-agent</identifier>
<location>
<url>https://aclanthology.org/2026.realm-1.47/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>687</start>
<end>701</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T What Agent Benchmarks Don’t Tell You: Recoverability and Credit Assignment in Agentic Retrieval Evaluation
%A Rallabandi, Sai Krishna
%A Dakle, Parag Pravin
%A Raghavan, Preethi
%Y Kamalloo, Ehsan
%Y Gontier, Nicolas
%Y Rozanov, Nikolai
%Y Hashemzadeh, Maryam
%Y Arora, Akhil
%S Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-485-9
%F rallabandi-etal-2026-agent
%X Agent quality on long-horizon tasks is usually reported as a single number: the aggregate gain of a tool-using agent over a non-agentic baseline on a benchmark. We argue that this number is not a property of the agent. It is jointly determined by a property of the benchmark and by which sub-component of the agent happens to be load-bearing, and reporting it alone makes agent quality look benchmark-dependent and irreproducible. We propose a four-stage evaluation protocol that separates the two. Stage 1 characterizes the benchmark before any outcome is observed, using an LLM-free diagnostic we call recoverability: whether evidence missed on the first retrieval is still reachable from what the system has already seen, and if so whether it is reachable by reformulation or only by structural navigation. Stage 2 assigns credit with an ablation ladder — one-shot RAG, a choice-free fixed-iterative loop, and a full agent — that localizes any measured gain to iteration or to open-ended agency. Stages 3 and 4 add the validity checks and cost accounting that change verdicts in practice. Applying the protocol to about 6,000 logged agent episodes over eight corpora and three reader families, we find that the headline agent gain ranges from \ensuremath-0.21 to +0.30 (each corpus in its native metric) purely as a function of the recoverability regime, and that on reformulation-recoverable corpora the choice-free loop recovers 65–96% of the gain normally credited to agency under the two stronger readers (and most of it under the open reader). Agency earns its cost only on the longest-horizon, navigation-recoverable corpus, where iteration saturates by round 2 and leaves a residual (+0.15 under the open reader at full N) that no round budget closes under any reader.
%U https://aclanthology.org/2026.realm-1.47/
%P 687-701
Markdown (Informal)
[What Agent Benchmarks Don’t Tell You: Recoverability and Credit Assignment in Agentic Retrieval Evaluation](https://aclanthology.org/2026.realm-1.47/) (Rallabandi et al., REALM 2026)
ACL