AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal


Abstract
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an automated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.
Anthology ID:
2026.findings-emnlp.598
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
12610–12639
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.598/
DOI:
Bibkey:
Cite (ACL):
Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. 2026. AgentRx: Diagnosing AI Agent Failures from Execution Trajectories. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 12610–12639, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories (Barke et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.598.pdf