Jongyeon Park

Author directory

2026

While Rotary Position Embedding (RoPE) is widely used in large language models, its inductive biases can impair long-context reasoning and lead to attention sinks. In this work, we propose TRONTOR, a method that effectively combines RoPE and No Positional Encoding (NoPE) through a simple fine-tuning phase, bypassing the need for expensive pretraining modifications. Our preliminary experiments demonstrate that mixing RoPE and NoPE at the attention-head level can enhance long-context capabilities, while static selection strategies fail to capture the varying complexity of inputs. Based on this observation, TRONTOR employs a lightweight context-aware routing method that dynamically switches each attention head between RoPE and NoPE at every token step. This router-only configuration provides an efficient operating point, yielding long-context performance gains with minimal inference overhead. For further gains, we extend TRONTOR to TRONTOR-Agg, which applies simple unweighted averaging over multiple RoPE bases within RoPE-active heads while avoiding learnable intra-head routing. Evaluated on the RULER benchmark using Llama-3.2 (1B and 3B) and Qwen3 (1.7B and 4B), TRONTOR improves long-context performance by mitigating attention sinks and dynamically reallocating attention toward relevant mid-context tokens.