As large language models are deployed at scale, the ability to selectively remove knowledge has become critical for privacy, safety, and legal compliance. Existing post-training unlearning methods suffer from a persistent tension: unbounded divergence signals over-unlearn, degrading utility and producing incoherent outputs, while fixed sample-agnostic targets and reference-relative penalties under-unlearn, leaving traces recoverable under paraphrase or membership-inference attacks. We attribute both failures to a common structural cause: the forgetting objective is decoupled from any sample- or parameter-level notion of what retention requires. We present FLOUR (Forgetting via Learned On-manifold Unlearning Representations), a latent unlearning framework in which a lightweight encoder produces a per-sample suppression direction. This direction is anchored to the null region of the reference model’s activations, keeping it on-manifold and disentangled from retention-critical directions, which enables forgetting while preserving coherent generation and enhancing privacy. Results on the Open-Unlearning benchmark suite show that FLOUR most closely matches the retrain oracle across the unlearning trilemma of forgetting, retention, and privacy, substantially outperforming prior baselines. Our code is publicly available at https://github.com/ judydnguyen/flour-unlearning.

Large language model context lengths have grown rapidly in recent years, from 512 tokens in GPT to 2M tokens in Gemini 1.5 Pro. Larger context windows enable models to condition on significantly more input tokens, leading to higher quality responses for some user prompts. However, longer contexts also pose challenges to system instruction adherence. In this work, we formalize verifiable instructions to evaluate model compliance based on clear, measurable criteria. From this criteria, we present VerIFY, a Verifiable Instruction Following Yardstick dataset designed to benchmark the compliance and accuracy of LLMs in adhering to various types of instructions across multi-turn, long-context conversations. From experiments with open-source models, we reveal insights into instruction-following failures in long contexts, helping to improve the reliability, safety, and precision of these models. Furthermore, we implement and evaluate six mitigation strategies to enhance instruction compliance in extended contexts, achieving an improvement up to 79%. This is the first work to consider instruction following for multi-turn, long context conversations.