Rongfeng Guo
Author directory2026
The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination
Wang Xi | Shijia Xu | Rongfeng Guo
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Wang Xi | Shijia Xu | Rongfeng Guo
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
While chain-of-thought prompting and scratchpads are frequently conceptualized as external memory for reasoning tasks, this perspective breaks down when a final answer must be generated from a bounded, jointly retained state. We formulate this limitation through a minimal constraint-satisfaction model in which an agent must exactly output N independent binary constraints using a shared internal and external memory configuration yielding at most 2B distinguishable states. Under this framework, we mathematically prove a tight phase law:
p⋆(N,B)=min{1,2B-N}.
This Joint Memory Budget Theorem establishes that each independent constraint exceeding the retained memory budget exponentially halves the probability of exact success. We further generalize this limit with adversarial robustness and a min-entropy extension that separates raw prompt length from effective independent information. To empirically test the qualitative predictions of this theory, we introduce a two-call retention protocol that forces large language models to reconstruct constraint hands solely from their bounded written representations. Comprehensive evaluations across modern long-context and reasoning models reveal sharp, capacity-dependent performance cliffs. While variables such as scratchpad formatting, effective task entropy, and answer-time external access can shift this phase boundary, they do not eliminate it. Ultimately, we demonstrate that long-context and scratchpad reasoning are fundamentally bottlenecked not by nominal context length, but by the independent information jointly available at the time of answer generation.
p⋆(N,B)=min{1,2B-N}.
This Joint Memory Budget Theorem establishes that each independent constraint exceeding the retained memory budget exponentially halves the probability of exact success. We further generalize this limit with adversarial robustness and a min-entropy extension that separates raw prompt length from effective independent information. To empirically test the qualitative predictions of this theory, we introduce a two-call retention protocol that forces large language models to reconstruct constraint hands solely from their bounded written representations. Comprehensive evaluations across modern long-context and reasoning models reveal sharp, capacity-dependent performance cliffs. While variables such as scratchpad formatting, effective task entropy, and answer-time external access can shift this phase boundary, they do not eliminate it. Ultimately, we demonstrate that long-context and scratchpad reasoning are fundamentally bottlenecked not by nominal context length, but by the independent information jointly available at the time of answer generation.
CentralKV: Compressing Long-Context LLMs via Attention-Graph Centrality
Yusen Wu | Huang Yinjun | Jia Yee Tan | Hao Li | Rongfeng Guo | Liang Fan | Jiachen Luo | Guangyuan Dong | Sike Xiang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yusen Wu | Huang Yinjun | Jia Yee Tan | Hao Li | Rongfeng Guo | Liang Fan | Jiachen Luo | Guangyuan Dong | Sike Xiang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Long-context inference in large language models is limited by the linear memory growth of the key-value (KV) cache. Existing eviction methods typically retain tokens with high accumulated attention or remove semantic redundancy, but may discard low-frequency tokens that connect distant dependencies. We propose CentralKV, a training-free framework that models attention as a dynamic sparse causal graph. Its Stream-Influence algorithm updates token centrality online and preserves structurally important bridge tokens during KV compression. The attention graph is constructed directly from the attention operation without separate similarity computation. Experiments on LongBench show that CentralKV improves over H2O, SnapKV, and GraphKV under strict physical memory constraints, particularly on reasoning-intensive tasks, with minimal latency overhead.