Akshey Sigdel

Author directory

2026

Reliable tool use is a fundamental capability for agentic systems, yet existing evaluations primarily focus on successful tool execution and provide limited insight into how agents recover from tool failures. To address this gap, we introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and fault-conditioned recovery. ToolMisuseBench evaluates recovery behavior across domains, fault types, and difficulty levels, using output-grounded success criteria and typed recoverability metadata. The seed-42 release includes 6,000 training, 900 development, and 1,200 public-test tasks covering CRUD, retrieval, file operations, and scheduling, including failures such as timeouts, rate limits, hidden first-call schema perturbations, authorization errors, and adversarial responses. Using the benchmark, we evaluate deterministic reference agents together with local 3-4B tool-calling LLMs (Qwen3-4B, Ministral-3B, and Llama-3.2-3B). On the public-test split, Parser+Repair achieves 0.767 task success and 0.736 conditional recovery success, compared with 0.158 task success when recovery is disabled. On a shared stratified 216-task diagnostic panel, Qwen3-4B, Ministral-3B, and Llama-3.2-3B achieve task success rates of 0.796, 0.903, and 0.773, respectively. Fault-level analysis reveals distinct recovery weaknesses that overall task success alone fails to capture. ToolMisuseBench provides a controlled and reproducible framework for diagnosing tool-use recovery under frozen fault plans, enabling more rigorous evaluation of robust tool-using agents beyond execution accuracy.