Jongyeon Park

Author directory

2026

While Rotary Position Embedding (RoPE) is widely used in large language models, its inductive biases can impair long-context reasoning and lead to attention sinks. In this work, we propose TRONTOR, a method that effectively combines RoPE and No Positional Encoding (NoPE) through a simple fine-tuning phase, bypassing the need for expensive pretraining modifications. Our preliminary experiments demonstrate that mixing RoPE and NoPE at the attention-head level can enhance long-context capabilities, while static selection strategies fail to capture the varying complexity of inputs. Based on this observation, TRONTOR employs a lightweight context-aware routing method that dynamically switches each attention head between RoPE and NoPE at every token step. This router-only configuration provides an efficient operating point, yielding long-context performance gains with minimal inference overhead. For further gains, we extend TRONTOR to TRONTOR-Agg, which applies simple unweighted averaging over multiple RoPE bases within RoPE-active heads while avoiding learnable intra-head routing. Evaluated on the RULER benchmark using Llama-3.2 (1B and 3B) and Qwen3 (1.7B and 4B), TRONTOR improves long-context performance by mitigating attention sinks and dynamically reallocating attention toward relevant mid-context tokens.
We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.