Linear attention architectures enable efficient long-context modeling by compressing sequence history into a fixed-size memory state. However, dense key value updates can overload the state as the context grows, causing memory collisions and degraded retrieval. Existing methods rely on memory decay, rank reduction, or heuristic routing to reduce interference, but they largely preserve dense online updates and do not explicitly control the structural complexity of memory. We introduce BregNet, a linear sequence architecture that formulates associative memory updates as a regularized online regression problem. Based on Linearized Bregman Iteration (LBI), BregNet decouples memory into a continuous dual accumulator and a structurally sparse primal state. The dual state records incoming information, while the sparse primal state keeps only important associations and removes noise through soft shrinkage. Experiments show that BregNet matches strong linear baselines in language modeling even with high memory sparsity. In this way, BregNet prevents the memory state from being overloaded by dense and interfering updates.