Mingyu Liu
Author directoryOther people with similar names: Mingyu Liu
Unverified author pages with similar names: Mingyu Liu
2026
TTT-Sparse: Test-Time Training of Sparse Attention for Long-Context Understanding
Xuehao Wang | Huayu Deng | Linhao Zhong | Yanbin Wei | Hengyu Zhao | Mingyu Liu | Hao Chen | Chunhua Shen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xuehao Wang | Huayu Deng | Linhao Zhong | Yanbin Wei | Hengyu Zhao | Mingyu Liu | Hao Chen | Chunhua Shen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Pretrained LLMs excel across diverse tasks, yet standard attention remains a major bottleneck for long-context understanding. Recent test-time training (TTT) methods alleviate this by encoding context into internal model parameters, but they typically require specialized architectures, meta-learned initialization, or continual post-training, preventing plug-and-play use on off-the-shelf LLMs. We propose test-time training sparse attention (TTT-Sparse), a plug-and-play alternative that performs test-time training over external memory while keeping the internal parameters frozen. For each input, TTT-Sparse constructs contextual chunk key states from the KV cache and learns a lightweight query adapter to induce a task-specific sparse attention pattern, which is used during generation to attend only to the most relevant context chunks and suppress interference from irrelevant context. Extensive experiments on long-context benchmarks show that TTT-Sparse improves long-context understanding while reducing generation-time attention cost.