EMER-Guard: Protocol-Grounded Verification for LLM Emergency Instructions

Zida Yang, Guquan You, Yu Zheng


Abstract
When LLM-generated emergency advice becomes part of public-facing information infrastructure, aggregate accuracy is no longer an adequate safety target: missing a hazardous instruction and blocking a correct one impose different human costs. We study protocol-grounded verification, the post-hoc auditing of emergency instructions against authoritative guidance under asymmetric risk. We introduce EMER-GuardBench, a 69,000-example counterfactual benchmark plus a 1,000-example benign-stress set spanning 10 disaster scenarios and 6 hazard mechanisms, with instruction-group-disjoint splits and cited protocol evidence. We further propose EGIS, an evidence-conditioned, protocol-supervised verifier that jointly predicts hazard status, hazard mechanism, severity, evidence support, and counterfactual ranking. This establishes a protocol-conditioned task and protocol-supervised representation learning, not inference-time causal grounding. EGIS reaches 89.9 Harm F1 with 10.4% IID over-refusal and zero benign-stress false positives; across 364 LLM-judge operating points, none Pareto-dominates EGIS on Harm F1, over-refusal, and severity-weighted miss rate, and a 14.4×-larger Qwen3-8B+LoRA supervised baseline still trails under identical splits. Ablations show that evidence helps as training-time supervision rather than prompt decoration. Our Severity-Weighted Miss Rate exposes a tradeoff hidden by scalar safety scores: zero missed severity can be achieved by broad refusal. EMER-Guard reframes emergency LLM safety from advice generation to auditable, protocol-grounded, cost-sensitive verification.
Anthology ID:
2026.emnlp-main.1205
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
27573–27593
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1205/
DOI:
Bibkey:
Cite (ACL):
Zida Yang, Guquan You, and Yu Zheng. 2026. EMER-Guard: Protocol-Grounded Verification for LLM Emergency Instructions. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 27573–27593, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
EMER-Guard: Protocol-Grounded Verification for LLM Emergency Instructions (Yang et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1205.pdf