Rongfeng Guo

Author directory

2026

While chain-of-thought prompting and scratchpads are frequently conceptualized as external memory for reasoning tasks, this perspective breaks down when a final answer must be generated from a bounded, jointly retained state. We formulate this limitation through a minimal constraint-satisfaction model in which an agent must exactly output N independent binary constraints using a shared internal and external memory configuration yielding at most 2B distinguishable states. Under this framework, we mathematically prove a tight phase law:

p⋆(N,B)=min{1,2B-N}.

This Joint Memory Budget Theorem establishes that each independent constraint exceeding the retained memory budget exponentially halves the probability of exact success. We further generalize this limit with adversarial robustness and a min-entropy extension that separates raw prompt length from effective independent information. To empirically test the qualitative predictions of this theory, we introduce a two-call retention protocol that forces large language models to reconstruct constraint hands solely from their bounded written representations. Comprehensive evaluations across modern long-context and reasoning models reveal sharp, capacity-dependent performance cliffs. While variables such as scratchpad formatting, effective task entropy, and answer-time external access can shift this phase boundary, they do not eliminate it. Ultimately, we demonstrate that long-context and scratchpad reasoning are fundamentally bottlenecked not by nominal context length, but by the independent information jointly available at the time of answer generation.
Long-context inference in large language models is limited by the linear memory growth of the key-value (KV) cache. Existing eviction methods typically retain tokens with high accumulated attention or remove semantic redundancy, but may discard low-frequency tokens that connect distant dependencies. We propose CentralKV, a training-free framework that models attention as a dynamic sparse causal graph. Its Stream-Influence algorithm updates token centrality online and preserves structurally important bridge tokens during KV compression. The attention graph is constructed directly from the attention operation without separate similarity computation. Experiments on LongBench show that CentralKV improves over H2O, SnapKV, and GraphKV under strict physical memory constraints, particularly on reasoning-intensive tasks, with minimal latency overhead.