Large language models (LLMs) are deployed with safety guardrails, yet jailbreak attacks and malicious prompts keep bypassing them. Re-aligning an LLM each time a new attack appears is more expensive than crafting attacks, so jailbreak detection is increasingly framed as a pre-processing step: a cheap detector filters out a malicious prompt before it reaches the LLM. This need is reinforced by multilingual attacks, where simply translating a prompt to another language can bypass the guardrails. Our goal is to find whether internal representations of a frozen LLM already separate malicious from benign prompts, and whether such a signal is resilient against translation. We study two types of detectors that require no fine-tuning of the LLM: a logistic-regression classifier over whole-sentence embeddings, and a probe-based detector that trains one logistic-regression probe per attention head on the attention activations of the last input token. They need at most a single forward pass. Our experiments show that both match or outperform strong sentence-embedding detectors and off-the-shelf baselines, specially under translation. The probes degrade more gracefully on higher-resource languages and remain competitive on lower-resource ones. Our results indicate that a frozen LLM linearly encodes whether an incoming prompt is malicious in a small set of mid-to-late layers. Per-head attention probes thus offer a defence that can be cheaply retrained as new attacks emerge, without touching the underlying model.

Natural language processing (NLP) models may leak private information in different ways, including membership inference, reconstruction, or attribute inference attacks. Sensitive information may not be explicit in the text, but hidden in underlying writing characteristics. Methods to protect privacy can involve using representations inside models that are demonstrated not to detect sensitive attributes or—for instance, in cases where users might be at risk from an untrustworthy model, the sort of scenario of interest here—changing the raw text before models can have access to it. The goal is to rewrite text to prevent someone from inferring a sensitive attribute (e.g., the gender of the author, or their location by the writing style) while keeping the text useful for its original intention (e.g., the sentiment of a product review). The few works tackling this have focused on generative techniques. However, these often create extensively different texts from the original ones or face problems such as mode collapse. This article explores a novel adaptation of adversarial attack techniques to manipulate a text to deceive a classifier w.r.t. one task (privacy) while keeping the predictions of another classifier trained for another task (utility) unchanged. We propose IDT, a method that analyses predictions made by auxiliary and interpretable models to identify which tokens are important to change for the privacy task, and which ones should be kept for the utility task. We evaluate different datasets for NLP suitable for different tasks. Automatic and human evaluations show that IDT retains the utility of text, while also outperforming existing methods when deceiving a classifier w.r.t. a privacy task.