Muthaheera Yasmeena Belgur Shamiullah
Author directory2026
Failing Agents Leave Traces: Telemetry Signatures for Multi-Tier Agent Evaluation
Muthaheera Yasmeena Belgur Shamiullah
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Muthaheera Yasmeena Belgur Shamiullah
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Agent evaluation today depends on per-trace LLM-judge inference or human review, too expensive to run on every trace; production systems fall back to sampling a fraction. We find that failing agents leave a detectable behavioral signature in standard observability telemetry: disproportionate effort relative to outcome. We formalize four behavioral failure signatures from agent telemetry, validated on 4,671 traces spanning five agent domains: Compensatory Effort (17–159% more effort on each domain’s primary signal, all five domains, p<0.006, the dominant signature every- where tested); Output Inflation (+38% larger patches on SWE-rebench; direction and reliability vary by domain); Unfocused Orchestra- tion (+5.98–12.3% higher token entropy, four of five domains); and Correlation Decoupling (effort-signal correlations shift structurally between pass and fail traces, p<0.001, three domains). A simple threshold check over these signals, with no judge, no label, and no model call, already catches 52.9–61.5% of failures. Across three human-labeled benchmarks, judges miss 6.5–18.1% of confirmed failures on the two directly comparable ones, and a trained telemetry classifier (AUROC 0.821) recovers 61.5–75.4% of each judge’s misses; combining judge and classifier cuts the miss rate from 18.1% to 4.4%, a 4.1× reduction against the weakest of the eight judges tested (GPT-4o-mini); stronger judges start with fewer misses, so the reduction is correspondingly smaller.