Long-context capability has become increasingly critical for next-generation Large Language Models (LLMs). However, the quadratic time complexity of standard softmax attention introduces substantial computation and memory overhead as sequence length grows, limiting the scalability of LLMs in long-context scenarios. Beyond existing approaches such as linear recurrent architectures and KV-cache compression methods, we propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts into learnable summary tokens. Specifically, summary tokens are periodically inserted into the input sequence and interleaved with text tokens, partitioning the sequence into distinct chunks while enabling text tokens and summary tokens to operate under different attention scopes. This design preserves dense local expressiveness through full attention among neighboring text tokens, while modeling long-range dependencies efficiently via summary tokens. Empirical results demonstrate that hybrid-KSA consistently outperforms existing hybrid attention variants in long-context modeling. Moreover, KSA is fully orthogonal to GQA and MLA during inference, and combining KSA with these methods achieves up to 8x KV-cache compression while preserving model performance.

Domain specialization can improve LLM behavior, but often weakens the general capabilities inherited from the original model. Recent Multi-Teacher On-Policy Distillation (MOPD) pipelines recover model capabilities by supervising student-generated trajectories with teacher feedback, but typically assume teacher-aligned prompt coverage, requiring prompts to match the teachers’ training distributions. This assumption is difficult to satisfy when the general teacher is an open-source model whose post-training data are unknown. Instead of attempting to reconstruct this hidden distribution, we study general capability recovery with readily available proxy general prompts. We identify two failure modes of vanilla MOPD in this incomplete-coverage situation: recovery-preservation counteraction from mixing conflicting recovery and preservation gradients, and weak-signal flattening from uniformly averaging samples with unequal correction demand. We propose **Counteraction-Aware Multi-Teacher On-Policy Distillation** (**CaMOPD**), which addresses these issues with decoupled alternating training and gap-based sample selection. CaMOPD allocates dedicated updates to general recovery, periodically performs domain-preservation updates, and selects samples with larger averaged token-level teacher-student log-probability gaps to concentrate correction signals. Across role-play dialogue and medical reasoning QA scenarios, CaMOPD outperforms all baselines in general capability recovery while maintaining domain-specific behavior. Gradient coherence analyses further support the intended effect of CaMOPD in producing more coherent correction signals.

Large language models (LLMs) exhibit powerful reasoning capacity, as evidenced by prior studies focusing on objective topics that with unique standard answers such as arithmetic and commonsense reasoning. However, the reasoning to definite answers emphasizes more on logical thinking, and falls short in effectively reflecting the comprehensive, reflective, and creative thinking that is also critical for the overall reasoning prowess of LLMs. In light of this, we build a dataset SJTP comprising diverse SubJective ToPics with free responses, as well as three evaluation indicators to fully explore LLM’s reasoning ability. We observe that a sole emphasis on logical thinking falls short in effectively tackling subjective challenges. Therefore, we introduce a framework grounded in the principle of the Negation of Negation (NeoN) to unleash the potential comprehensive, reflective, and creative thinking abilities of LLMs. Comprehensive experiments on SJTP demonstrate the efficacy of NeoN, and the enhanced performance on various objective reasoning tasks unequivocally underscores the benefits of stimulating LLM’s subjective thinking in augmenting overall reasoning capabilities.