Allan Kazakov

Author directory

2026

RoPE is a standard positional encoding for modern language models, but its fixed rotations can fail to distinguish positions and tokens in long contexts. Content-aware rotary methods such as CARoPE suggest that adapting rotary geometry to token content can improve language modeling. We report a negative-result study of four token-conditioned RoPE variants for language-model pretraining. DiPE adds token-dependent endpoint phase offsets; Gated DiPE bounds these offsets with learned gates; HSA-RoPE and PPA-RoPE instead modulate accumulated rotary frequency increments at the head and rotary-pair levels. Although these methods produce promising short-context gains, their improvements do not scale reliably. DiPE improves substantially at sequence length 1024 but weakens with larger models and longer contexts, while Gated DiPE fails to recover the lost benefit. HSA-RoPE and PPA-RoPE outperform RoPE at all tested sequence lengths, but their advantage shrinks with longer contexts and they fall substantially behind CARoPE at 8192 tokens.
Search
Co-authors
    Venues
    Fix author