Narjes Sharafi
Author directory2026
Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes
Ali Satvaty | Narjes Sharafi | Jirui Qi | Suzan Verberne | Fatih Turkmen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ali Satvaty | Narjes Sharafi | Jirui Qi | Suzan Verberne | Fatih Turkmen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models (LLMs) can expose memorized training sequences under prefix-based extraction: given a prefix from a training example, the model may assign high probability to the original continuation. In deployed systems, however, prefixes are rarely evaluated in isolation. They often appear together with instructions, retrieved documents, or other task-specific context, as in retrieval-augmented generation (RAG). This raises a basic question: does contextual conditioning reduce memorization, or does it change which memorized samples become exposed? We study this question through paired item-level measurements of probabilistic suffix extraction. For each prefix-suffix pair, we score the same target suffix under an empty prompt and under retrieved contexts of varying relevance, across three open-weight instruction-tuned models. We find that context does not simply erase memorization. Instead, extractable memorization consists of a context-robust core and a context-sensitive boundary. Many samples that are extractable without context remain extractable under retrieved context, especially as prefix length increases. At the same time, context mainly affects marginal samples near the extraction threshold: it suppresses some exposures, but also enables new ones that are missed by prefix-only evaluation. These findings qualify the view that RAG reduces memorization risk. Context can lower aggregate extraction by suppressing boundary cases, yet robustly extractable samples persist, and context-enabled extractability remains security-relevant.