Shinda Huang
Author directory2026
SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints
Zekun Li | Shinda Huang | Jiangtian Wang | Nathan Zhang | Antonis Antoniades | Wenyue Hua | Kaijie Zhu | Sirui Zeng | Chi Wang | William Yang Wang | Xifeng Yan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zekun Li | Shinda Huang | Jiangtian Wang | Nathan Zhang | Antonis Antoniades | Wenyue Hua | Kaijie Zhu | Sirui Zeng | Chi Wang | William Yang Wang | Xifeng Yan
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
As language agents are increasingly entrusted with tool-mediated actions, reaching the correct outcome is not sufficient: agents must also comply with domain-specific standard operating procedures (SOPs), policies, and constraints throughout execution. We introduce SOPBench, an executable benchmark for evaluating procedural compliance in language agents. SOPBench provides sandbox environments spanning seven customer-service domains, with 167 executable tools, 70 service-specific SOPs, and 830 validated test cases. Its code-based oracle verifiers assess compliance at the outcome, step, and trajectory levels without relying on human or LLM judges. Evaluating 23 leading models, we find that reasoning models perform best but remain far from reliable, particularly in difficult domains and under simple adversarial instructions. Error analysis further reveals distinct procedural failure modes across model capability levels. Finally, a layered harness ablation shows that even when agents are given ground-truth constraint verdicts, 12-38% of cases still fail. This residual gap suggests that a key challenge lies not only in retrieving or verifying constraints, but also in consistently translating verified constraints into compliant actions. We release the code, data, and more than 40k agent trajectories.