Xiaoyu Wang
Author directoryOther people with similar names: Xiaoyu Wang, Xiaoyu Wang, Xiaoyu Wang, Xiaoyu Wang
Unverified author pages with similar names: Xiaoyu Wang
2026
Beyond Single-Policy: Evaluating Composed Organization-Specific Policy Alignment in LLM Chatbots
Yingjie Liu | Yongxiang Hu | Xuan Wang | Yilun Li | Yunlei Wei | Xiaoyu Wang | Yangfan Zhou
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yingjie Liu | Yongxiang Hu | Xuan Wang | Yilun Li | Yunlei Wei | Xiaoyu Wang | Yangfan Zhou
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language model chatbots are increasingly deployed in organizational settings such as healthcare, finance, and public services, where organization-specific policies specify allowed and prohibited content. Existing policy-alignment benchmarks construct tests from one policy at a time, leaving failures caused by policy composition under-tested. We present COPAL, an automated framework for evaluating composed-policy alignment in organizational chatbots. COPAL uses empirically derived interaction patterns to generate queries that require multiple policies to be handled in one response, and pairs each query with an explicit handling contract specifying what to provide and avoid. Applied to 30 organization-like company worlds, COPAL exposes substantial composed-policy handling failures. Across 9 served models in our automated evaluation setup, composed-policy requests yield a 33.1% error rate, indicating that composed-policy alignment remains a challenging evaluation target. Our code is available at https://github.com/BruceAllenCS/COPAL-code.