Mingyu Liu

Author directory

Other people with similar names: Mingyu Liu

Unverified author pages with similar names: Mingyu Liu


2026

Pretrained LLMs excel across diverse tasks, yet standard attention remains a major bottleneck for long-context understanding. Recent test-time training (TTT) methods alleviate this by encoding context into internal model parameters, but they typically require specialized architectures, meta-learned initialization, or continual post-training, preventing plug-and-play use on off-the-shelf LLMs. We propose test-time training sparse attention (TTT-Sparse), a plug-and-play alternative that performs test-time training over external memory while keeping the internal parameters frozen. For each input, TTT-Sparse constructs contextual chunk key states from the KV cache and learns a lightweight query adapter to induce a task-specific sparse attention pattern, which is used during generation to attend only to the most relevant context chunks and suppress interference from irrelevant context. Extensive experiments on long-context benchmarks show that TTT-Sparse improves long-context understanding while reducing generation-time attention cost.