Carolina Ternero
Author directory2026
When Harmful Intent Lies Between the Lines: Pragmatics-Based Automated Red Teaming for LLM Safety Evaluation
Shuying Ye | Carolina Ternero | Fei Xia
Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)
Shuying Ye | Carolina Ternero | Fei Xia
Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH)
As large language models (LLMs) are increasingly deployed in real-world applications, ensuring their safety is a critical concern. Most safety evaluations focus on direct harmful requests, overlooking cases where the same harmful intent is disguised through indirect, polite, hypothetical, or contextually framed language. To address this limitation, we propose a pragmatics-based automated red-teaming framework that tests whether LLM safety behavior changes when malicious intent is repackaged through different pragmatic strategies. First, we develop a linguistically grounded taxonomy of seven pragmatic strategies—presupposition, implicature, indirect speech acts, denial disclaimers, politeness, hypothetical framing, and contextual framing—and use it to generate adversarial prompts across seven harm categories. Second, we evaluate these prompts on three Target Models (GPT-4o-mini, Qwen2.5-7B, Llama-3.3-70B), with Claude Haiku as judge and human validation. Our experiments show that pragmatically transformed prompts consistently yield higher Attack Success Rate (ASR) than original seed prompts, with contextual framing emerging as particularly effective. These results suggest LLM safety evaluation should account for pragmatic meaning and inferred intent, not just explicit harmful content.