Tian Zihan

Author directory

2026

Restricting attention to a sliding window makes long documents affordable, but it discards everything beyond the window. This paper asks what can be recovered when the model is already trained, its weights are frozen, and the budget for new parameters is only a few tens of thousands. We propose Multi-scale Spectral Residual Attention (MSRA): a far-field branch grafted onto the last few layers of a frozen model, which condenses out-of-window history through smoothly distance-decayed weights and adds it back to the attention output—roughly 9K–45K trainable parameters, plus minimal 8-to-16-scalar probes for attribution. On PG-19 long-form fiction, the branch closes up to 31.7% of the perplexity gap between local and full attention, with gains stable across two architectures, multiple seeds, and the official test partition. The attribution proves sharper than the headline: 16 learnable output-gain scalars alone close 26.4%, returns accelerate with grafting depth, the carefully designed kernel structure and gating contribute only secondarily—and a budget-matched LoRA closes 87.4%, indicating that the gap is mostly generic capacity rather than a missing far-field pathway. Retrieval experiments complete the map: the correction is a statistical smoother that helps just beyond the window, slightly hurts precise in-window prediction, recovers essentially nothing at genuine long distance, and does not transfer to task metrics—while slowing prefill two-to-three-fold and leaving decoding unaffected. These results bound what a lightweight spectral correction can and cannot recover, forming a reproducible decision reference for local long-context design.
Search
Co-authors
    Venues
    Fix author