Large Language Models (LLMs) have achieved strong performance across diverse NLP tasks, but deploying them in real-world scenarios raises concerns regarding safety, reliability, and computational cost. To mitigate the risk of harmful, unreliable, or unfaithful content, LLM pipelines increasingly rely on guardrails. Yet deploying multiple guardrails across different tasks and risk categories introduces significant computational and orchestration overhead. In this work, we propose a calibrated deferral-routing framework for deploying guardrails both accurately and cost-effectively. Our approach arranges guardrails as a cascade of increasing capability and cost: each input is first evaluated by a lightweight guardrail, while only uncertain instances are deferred to a larger, more capable, and more expensive model. We formalize this routing problem and study practical deferral-calibration strategies by applying post-hoc calibration, including isotonic regression, Platt scaling, and temperature scaling. We evaluate 18 guardrail models from four model families (i.e., Granite Guardian, Llama Guard, ShieldGemma, and GuardReasoner) across 10 harm-detection benchmarks. Our experiments show that: (i) all guardrail models are often poorly calibrated out of the box (ECE ranging from 6% to 28%); (ii) post-hoc calibration substantially improves reliability reducing ECE below 2.5% in most cases; (iii) cascade routing yields large cost savings with minimal accuracy loss, while monotone calibration primarily makes routing thresholds deployable rather than improving rank ordering.

The deployment of language models in real-world applications exposes users to various risks, including hallucinations and harmful or unethical content. These challenges highlight the urgent need for robust safeguards to ensure safe and responsible AI. To address this, we introduce Granite Guardian, a suite of advanced models designed to detect and mitigate risks associated with prompts and responses, enabling seamless integration with any large language model (LLM). Unlike existing open-source solutions, our Granite Guardian models provide comprehensive coverage across a wide range of risk dimensions, including social bias, profanity, violence, sexual content, unethical behavior, jailbreaking, and hallucination-related issues such as context relevance, groundedness, and answer accuracy in retrieval-augmented generation (RAG) scenarios. Trained on a unique dataset combining diverse human annotations and synthetic data, Granite Guardian excels in identifying risks often overlooked by traditional detection systems, particularly jailbreak attempts and RAG-specific challenges. https://github.com/ibm-granite/granite-guardian

Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated action. In this paper, we investigate the underlying mechanisms of these attacks by analyzing the attention patterns within LLMs. We introduce the concept of the distraction effect, where specific attention heads, termed important heads, shift focus from the original instruction to the injected instruction. Building on this discovery, we propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks without the need for additional LLM inference. Our method generalizes effectively across diverse models, datasets, and attack types, showing an AUROC improvement of up to 10.0% over existing methods, and performs well even on small LLMs. We demonstrate the robustness of our approach through extensive evaluations and provide insights into safeguarding LLM-integrated systems from prompt injection vulnerabilities.

The wide applicability and adaptability of generative large language models (LLMs) has enabled their rapid adoption. While the pre-trained models can perform many tasks, such models are often fine-tuned to improve their performance on various downstream applications. However, this leads to issues over violation of model licenses, model theft, and copyright infringement. Moreover, recent advances show that generative technology is capable of producing harmful content which exacerbates the problems of accountability within model supply chains. Thus, we need a method to investigate how a model was trained or a piece of text was generated and what their pre-trained base model was. In this paper we take the first step to address this open problem by tracing back the origin of a given fine-tuned LLM to its corresponding pre-trained base model. We consider different knowledge levels and attribution strategies, and find that we can correctly trace back 8 out of the 10 fine tuned models with our best method.