Recoverability-Controlled Safety Miscalibration in Large Language Models

Yuwei Wang, Xingwang Liang, Yu Xie, Haotian Chi, Haijun Geng


Abstract
Safety-aligned language models are expected to refuse harmful requests while remaining useful for benign ones. However, because many evaluations score only the final response, they often conflate intent understanding with response-level safety behavior. To address this, we introduce RiskCueBench, a recoverability-controlled benchmark that separates these two stages by first measuring whether a model can successfully recover a high-level intent and then evaluating its safety behavior on the same prompt. RiskCueBench contains 160 latent intents and 640 abstract non-operational surface prompts across eight risk categories. The benchmark covers 25 completed model groups and 32,000 successful model outputs with final rubric-adjudicated labels. When conditioning on pairs where both surface forms are recoverable, we find that increased contextualized specificity induces a shift toward less safe behavior. Specifically, paired bootstrap estimates show that the refusal rate and global safety score decrease by 0.175 and 0.440, respectively. Furthermore, mixed effects modeling corroborates a 0.107 increase in unsafe cooperation. These results show that the safety behavior can be miscalibrated to the surface form even after the intent has been recovered, motivating safety evaluations that report response behavior conditional on semantic recoverability.
Anthology ID:
2026.emnlp-main.630
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
14463–14481
Language:
URL:
https://aclanthology.org/2026.emnlp-main.630/
DOI:
Bibkey:
Cite (ACL):
Yuwei Wang, Xingwang Liang, Yu Xie, Haotian Chi, and Haijun Geng. 2026. Recoverability-Controlled Safety Miscalibration in Large Language Models. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 14463–14481, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Recoverability-Controlled Safety Miscalibration in Large Language Models (Wang et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.630.pdf