Yixue Hao
Author directory2026
TAKE: Task-Aware Chunked KV Cache Eviction for Efficient Long-Context LLM Prefill
Rui Wang | Nan Jia | Jiahui Li | Qingyi Yang | Yixue Hao | Long Hu
Findings of the Association for Computational Linguistics: EMNLP 2026
Rui Wang | Nan Jia | Jiahui Li | Qingyi Yang | Yixue Hao | Long Hu
Findings of the Association for Computational Linguistics: EMNLP 2026
Memory bottlenecks severely constrain the deployment of large language models (LLMs) in long-context scenarios. While existing global pruning methods reduce memory usage during the decoding phase, they largely ignore out-of-memory errors in the prefill stage before generation begins. To address this, we present Task-Aware Chunked KV Cache Eviction (TAKE), a training-free framework for memory-efficient long-context prefill. TAKE refines the prefill process by partitioning long sequences into chunks and performing incremental KV eviction, thereby avoiding the computational overhead of full-sequence processing. Crucially, to mitigate the semantic discontinuity introduced by chunk-wise processing, we introduce accumulated task-aware probe tokens and a delayed eviction strategy to reduce information loss. Extensive evaluations show that TAKE achieves the highest efficiency, slashing peak memory usage to 59.53% of the baseline while accelerating Time-To-First-Token (TTFT) by over 60% under 128k-token inference. Our code is available at [Source Code of TAKE](https://github.com/ObitoJN/TAKE).
Revealing Procedural Reasoning Structures in Chain-of-Thought Training via Span-Level Gradient Organization
Jia Liu | Jiaxin Luo | Weiwen Xu | Jonathan M. Garibaldi | Xiao-Kun Wu | Yixue Hao | Min Chen
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Jia Liu | Jiaxin Luo | Weiwen Xu | Jonathan M. Garibaldi | Xiao-Kun Wu | Yixue Hao | Min Chen
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Chain-of-Thought (CoT) prompting enables large language models to produce multi-step reasoning, yet how such reasoning-related structure is expressed during training remains poorly understood. We present Gradient-based Structural Developer (GSD), an unsupervised framework with a principled gradient aggregation view that tracks span-level gradient during fine-tuning on reasoning benchmarks to understand how models develop structured, step-by-step reasoning capabilities. Our analysis shows that while gradients at the level of individual tokens are often noisy, aggregating gradients over contiguous reasoning-related spans reveals stable and recurring directional alignment across samples. We refer to these directionally aligned patterns as aligned sequential stresses, reflecting consistent gradient organization associated with similar reasoning procedures. Beyond capturing semantically similar reasoning instances, such gradient alignment also reveals structurally similar but semantically diverse cases that share common procedural organization. These findings position GSD as a diagnostic framework for analyzing how procedural reasoning structures emerge during training, with downstream selection results serving as auxiliary evidence correlating gradient alignment with adaptation efficiency.