Jiangtian Wang

Author directory

2026

As language agents are increasingly entrusted with tool-mediated actions, reaching the correct outcome is not sufficient: agents must also comply with domain-specific standard operating procedures (SOPs), policies, and constraints throughout execution. We introduce SOPBench, an executable benchmark for evaluating procedural compliance in language agents. SOPBench provides sandbox environments spanning seven customer-service domains, with 167 executable tools, 70 service-specific SOPs, and 830 validated test cases. Its code-based oracle verifiers assess compliance at the outcome, step, and trajectory levels without relying on human or LLM judges. Evaluating 23 leading models, we find that reasoning models perform best but remain far from reliable, particularly in difficult domains and under simple adversarial instructions. Error analysis further reveals distinct procedural failure modes across model capability levels. Finally, a layered harness ablation shows that even when agents are given ground-truth constraint verdicts, 12-38% of cases still fail. This residual gap suggests that a key challenge lies not only in retrieving or verifying constraints, but also in consistently translating verified constraints into compliant actions. We release the code, data, and more than 40k agent trajectories.