Fengyuan Fan
Author directory2026
REFOCUS: Training-Free Attention Sparsification via Uniform-Reference Filtering
Fengyuan Fan | Rena Wei Gao | Yanbei Jiang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Fengyuan Fan | Rena Wei Gao | Yanbei Jiang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Dense softmax attention assigns non-zero probability mass to many task-irrelevant keys, which can dilute useful signals in the attention output. We present REFOCUS, a training-free and plug-and-play inference-time sparsification method that keeps attention entries above a row-wise, context-length-aware threshold and renormalizes the remaining weights. This design is stable, simple to deploy, and requires no model retraining. REFOCUS is integrated in HuggingFace via attn_implementation="refocus" without model-code modification. On Qwen3-4B under 0-shot greedy decoding, REFOCUS matches or outperforms dense eager attention on full-test GSM8K, HumanEval, WinoGrande, and AQuA-RAT, and improves RULER-4k by +0.52 percentage points overall, including +3.20 points on aggregation tasks. These results indicate that a single row-wise threshold can remove diffuse long-tail attention while preserving or improving accuracy.