Zhaolong Su

Author directory

2026

Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost, resulting in expensive inference and an inability to revise previous tokens during text generation. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent and underdeveloped. In this work, we propose B3D-RWKV, a diffusion-based RWKV variant that integrates the model’s 𝒪(L) inference efficiency with parallel, bidirectional discrete-diffusion through a triplet-block layout method. B3D-RWKV-7.2B reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of 1.6× speedup. Code is available at https://github.com/leonardodalinky/B3D-RWKV.