Retrieval-Augmented Generation (RAG) mitigates hallucinations by grounding generation in retrieved evidence, yet failures persist when models underutilize the retrieved context. While prior work attributes this to conflicts between external context and internal parametric knowledge encoded in Feed-Forward Networks (FFNs), our empirical analyses reveal that FFN-derived signals are unstable and insufficient for reliable diagnosis across diverse contexts. We therefore propose the Attention Escape Score (AES), a training-free diagnostic framework that provides a mechanistic lens on RAG hallucinations via dynamic attention routing failures. We models the failures as weakened Semantic Anchoring, increased Contextual Distraction, and heightened Generation Inertia, moving beyond static component-level analyses. Extensive experiments show that AES significantly outperforms state-of-the-art baselines with robust cross-model generalization. Furthermore, we introduce Pruning Escape Routes (PER), a lightweight inference-time intervention that suppresses hallucinations by attenuating identified escape pathways, achieving improved faithfulness with negligible degradation to complex reasoning and general capabilities.

Advanced LLMs have achieved near-ceiling instruction-following accuracy on benchmarks such as IFEval. However, these impressive scores do not necessarily translate to reliable services in real-world use, where users often vary their phrasing, contextual framing, and task formulations. In this paper, we study nuance-oriented reliability: whether models exhibit consistent competence across cousin prompts that convey analogous user intents but with subtle nuances. To quantify this, we introduce a new metric, reliable@k, and develop an automated pipeline that generates high-quality cousin prompts via data augmentation. Building upon this, we construct IFEval++ for systematic evaluation. Across 20 proprietary and 26 open-source LLMs, we find that current models exhibit substantial insufficiency in nuance-oriented reliability—their performance can drop by up to 61.8% with nuanced prompt modifications. What’s more, we characterize it and explore three potential improvement recipes. Our findings highlight nuance-oriented reliability as a crucial yet underexplored next step toward more dependable and trustworthy LLM behavior. Our code and benchmark are accessible: https://github.com/jianshuod/IFEval-pp.

Large language models (LLMs) exhibit prompt leakage vulnerabilities, where they may be coaxed into revealing system prompts embedded in LLM services, raising intellectual property and confidentiality concerns. An intriguing question arises: Do LLMs genuinely internalize prompt leakage intents in their hidden states before generating tokens? In this work, we use probing techniques to capture LLMs’ intent-related internal representations and confirm that the answer is yes. We start by comprehensively inducing prompt leakage behaviors across diverse system prompts, attack queries, and decoding methods. We develop a hybrid labeling pipeline, enabling the identification of broader prompt leakage behaviors beyond mere verbatim leaks. Our results show that a simple linear probe can predict prompt leakage risks from pre-generation hidden states without generating any tokens. Across all tested models, linear probes consistently achieve 90%+ AUROC, even when applied to new system prompts and attacks. Understanding the model internals behind prompt leakage drives practical applications, including intention-based detection of prompt leakage risks. Code is available at: https://github.com/jianshuod/Probing-leak-intents.