2026

Current evaluations of memorization in Large Language Models (LLMs) predominantly rely on overlap ratios, which fails to distinguish between sporadic token matches and reliable, sequential recall. What’s more, these methods become inapplicable in practical scenarios where ground truth suffixes is unavailable. To address these limitations, we propose a paradigm shift from quantifying "rates" to determining the Effective Memorization Length—the distinct boundary where a model transitions from faithful recall to open-ended generation. We introduce a Ensemble Memorization Classifier that leverages internal states from the LLM to estimate this boundary in a reference-free manner. This framework enables Confidence-Aware Generation Control, allowing to extract high-precision suffixes by dynamically truncating sequences precisely at the memorization boundary. Furthermore, we demonstrate that this boundary is optimizable; by integrating our classifier as a reward model within a reinforcement learning framework, we actively extend the average effective length of generated suffixes. Experiments on the GPT-Neo and Pythia families demonstrate that our approach not only accurately filters non-memorization tokens but also effectively "unlocks" deeper chains of memorized knowledge, transforming memorization from an unpredictable risk into a controllable utility.

2015