Bingxuan Hou


2025

pdf bib
Disentangle to Decay: Linear Attention with Trainable Decay Factor
Haibo Tong | Chenyang Zhang | Jiayi Lin | Bingxuan Hou | Qingqing Hong | Junli Wang
Proceedings of the 31st International Conference on Computational Linguistics

Linear attention enhances inference efficiency of Transformer and has attracted research interests as an efficient backbone of language models. Existing linear attention based models usually exploit decay factor based positional encoding (PE), where attention scores decay exponentially with increasing relative distance. However, most work manually designs a non-trainable decay factor of exponential calculation, which limits further optimization. Our analysis reveals directly training decay factor is unstable because of large gradients. To address this, we propose a novel PE for linear attention named Disentangle to Decay (D2D). D2D disentangles decay factor into two parts to achieve further optimization and stable training. Moreover, D2D can be transformed into recurrent form for efficient inference. Experiments demonstrate that D2D achieves stable training of decay factor, and enhances performance of linear attention in both normal context length and length extrapolation scenarios.

2024

pdf bib
Correct after Answer: Enhancing Multi-Span Question Answering with Post-Processing Method
Jiayi Lin | Chenyang Zhang | Haibo Tong | Dongyu Zhang | Qingqing Hong | Bingxuan Hou | Junli Wang
Findings of the Association for Computational Linguistics: EMNLP 2024