Zhiyuan Liang
Author directory2026
Kwai Summary Attention
Chenglong Chu | Hao Peng | Hui Wang | Jiangxia Cao | Lu Ren | Zhiyuan Liang | Ziqi Wang | Hongtao Cheng | Yi Su | Xinchen Luo | Jian Liang | Ruiming Tang | Guorui Zhou | Han Li
Findings of the Association for Computational Linguistics: EMNLP 2026
Chenglong Chu | Hao Peng | Hui Wang | Jiangxia Cao | Lu Ren | Zhiyuan Liang | Ziqi Wang | Hongtao Cheng | Yi Su | Xinchen Luo | Jian Liang | Ruiming Tang | Guorui Zhou | Han Li
Findings of the Association for Computational Linguistics: EMNLP 2026
Long-context capability has become increasingly critical for next-generation Large Language Models (LLMs). However, the quadratic time complexity of standard softmax attention introduces substantial computation and memory overhead as sequence length grows, limiting the scalability of LLMs in long-context scenarios. Beyond existing approaches such as linear recurrent architectures and KV-cache compression methods, we propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts into learnable summary tokens. Specifically, summary tokens are periodically inserted into the input sequence and interleaved with text tokens, partitioning the sequence into distinct chunks while enabling text tokens and summary tokens to operate under different attention scopes. This design preserves dense local expressiveness through full attention among neighboring text tokens, while modeling long-range dependencies efficiently via summary tokens. Empirical results demonstrate that hybrid-KSA consistently outperforms existing hybrid attention variants in long-context modeling. Moreover, KSA is fully orthogonal to GQA and MLA during inference, and combining KSA with these methods achieves up to 8x KV-cache compression while preserving model performance.
GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs
Weidong Tang | Jierui Li | Yueling Hou | Zihan Mei | Can Zhang | Xinyan Wan | Zhiyuan Liang | Pengfei Zhou | Yang You | Wangbo Zhao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Weidong Tang | Jierui Li | Yueling Hou | Zihan Mei | Can Zhang | Xinyan Wan | Zhiyuan Liang | Pengfei Zhou | Yang You | Wangbo Zhao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
True general intelligence requires not only a model of the physical world but also a social world model: the capacity to infer how individual mental states interact and crystallize into group-level outcomes. Despite notable progress in individual-level Theory of Mind (ToM) reasoning, existing multimodal large language models systematically fail at this: collective behavior emerges non-linearly from social tensions, conformity dynamics, and structural constraints, and cannot be recovered by summing individual intentions. We present GroupToM-Bench, the first multimodal benchmark for group-level ToM, built around a causal chain spanning micro-level BDI states (belief, desire, intention), meso-level group tension and structural constraints, and macro-level outcome prediction and mechanistic attribution. To probe this full arc, we develop a seven-level cognitive audit framework. Experiments reveal that frontier models perform significantly below human levels, exposing fundamental blind spots in modeling social structures and nonlinear collective behavior.