Srinivas Harish
Author directory2026
Two Thirds of the Best-of-K Oracle Band in Tool Agents Is Unreachable by Reward-Free Selection
Srinivas Harish
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Srinivas Harish
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Running a tool agent n times and keeping the best run is worth 2.19–22.37 points on τ²-bench, a floor any scaffold spending n times the calls has to clear. Choosing among those runs, for no additional model calls, is worth anywhere from −23.68 to +12.28 depending on which rule does the choosing. We evaluate 9 selectors that read only the agent’s own trajectory across 28 model × domain cells and 10 403 released trials, and add an LLM judge that reads the trajectories but never a reward and never the evaluation criteria. Every arm is therefore reward-free: the nine rules are verifier-free as well, the judge is not, and neither sees a label. Against a mean best-of-K oracle band of 12.29 points, which is what a selector holding the reward would collect, the judge recovers 32.7% and the best free rule 21.0%, leaving 67.3% of that band unreachable without the label. One measurable property of the rollouts orders the task distributions, the no-mode rate, the share of tasks on which no two of the K rollouts emit the same write-action sequence, and what it predicts is when agreement is exploitable rather than when selection is possible. Where agreement-based rules are nearly inert the judge is worth +5.95 against +0.53; where the rollouts are undifferentiated it returns 1.19 against a band of 14.4 points; on a third domain it returns −0.64 and loses to a free heuristic costing one fewer call.