While Transformer models can learn algorithmic tasks and generalize reliably to unseen, in-distribution data, they often fail catastrophically when required to extrapolate to sequence lengths beyond their training regime. Using AttentionSpan, a suite of synthetic diagnostic tasks designed to enable inspection of internal model computations, we analyze the model’s behavior on length extrapolation. Our findings indicate that this failure is not fully explained by an inability to induce general rules for the task. Instead, we identify inconsistent attention patterns as a major contributing factor: information retrieval strategies learned by individual attention heads, which fail to remain stable as sequence length increases. Reinforcing a reference attention pattern at inference time, using no out-of-distribution data, largely recovers accuracy on String Reversal and partially on Value Assignment, while leaving multi-digit arithmetic unresolved. We also study length adaptation, how a model trained on one length adjusts to another by fine-tuning its Key and Query parameters, and find that a single dimension in every attention head is enough, across 4 algorithmic tasks and the RULER benchmark. Adaptation does not carry past the fine-tuned lengths, and our interventions are diagnostic rather than a proposed remedy. We make our benchmark and code publicly available.

This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding’s theoretical limitations. Through an analysis spanning model scales, architectures, and training checkpoints, we show that attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. We demonstrate that this scale-invariant behavior is unique to RoPE, emerges through distinct evolutionary phases during training, and statistically adheres to the fundamental uncertainty principle. Our findings suggest that the effectiveness of modern Transformers stems from their remarkable ability to spontaneously develop optimal, multi-resolution decompositions to address inherent architectural constraints.