<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kazakov-2026-short">
    <titleInfo>
        <title>Short-Context Gains, Long-Context Drift: Negative Results for Token-Conditioned RoPE</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Allan</namePart>
        <namePart type="family">Kazakov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Seventh Workshop on Insights from Negative Results in NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">João</namePart>
            <namePart type="family">Sedoc</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raphael</namePart>
            <namePart type="family">Shu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aleksandr</namePart>
            <namePart type="family">Drozd</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Arjun</namePart>
            <namePart type="family">Akula</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Salvatore</namePart>
            <namePart type="family">Giorgi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Adrian</namePart>
            <namePart type="family">Benton</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-480-4</identifier>
    </relatedItem>
    <abstract>RoPE is a standard positional encoding for modern language models, but its fixed rotations can fail to distinguish positions and tokens in long contexts. Content-aware rotary methods such as CARoPE suggest that adapting rotary geometry to token content can improve language modeling. We report a negative-result study of four token-conditioned RoPE variants for language-model pretraining. DiPE adds token-dependent endpoint phase offsets; Gated DiPE bounds these offsets with learned gates; HSA-RoPE and PPA-RoPE instead modulate accumulated rotary frequency increments at the head and rotary-pair levels. Although these methods produce promising short-context gains, their improvements do not scale reliably. DiPE improves substantially at sequence length 1024 but weakens with larger models and longer contexts, while Gated DiPE fails to recover the lost benefit. HSA-RoPE and PPA-RoPE outperform RoPE at all tested sequence lengths, but their advantage shrinks with longer contexts and they fall substantially behind CARoPE at 8192 tokens.</abstract>
    <identifier type="citekey">kazakov-2026-short</identifier>
    <location>
        <url>https://aclanthology.org/2026.insights-1.13/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>102</start>
            <end>109</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Short-Context Gains, Long-Context Drift: Negative Results for Token-Conditioned RoPE
%A Kazakov, Allan
%Y Sedoc, João
%Y Shu, Raphael
%Y Drozd, Aleksandr
%Y Akula, Arjun
%Y Rogers, Anna
%Y Giorgi, Salvatore
%Y Benton, Adrian
%S Proceedings of the Seventh Workshop on Insights from Negative Results in NLP
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-480-4
%F kazakov-2026-short
%X RoPE is a standard positional encoding for modern language models, but its fixed rotations can fail to distinguish positions and tokens in long contexts. Content-aware rotary methods such as CARoPE suggest that adapting rotary geometry to token content can improve language modeling. We report a negative-result study of four token-conditioned RoPE variants for language-model pretraining. DiPE adds token-dependent endpoint phase offsets; Gated DiPE bounds these offsets with learned gates; HSA-RoPE and PPA-RoPE instead modulate accumulated rotary frequency increments at the head and rotary-pair levels. Although these methods produce promising short-context gains, their improvements do not scale reliably. DiPE improves substantially at sequence length 1024 but weakens with larger models and longer contexts, while Gated DiPE fails to recover the lost benefit. HSA-RoPE and PPA-RoPE outperform RoPE at all tested sequence lengths, but their advantage shrinks with longer contexts and they fall substantially behind CARoPE at 8192 tokens.
%U https://aclanthology.org/2026.insights-1.13/
%P 102-109

Markdown (Informal)

[Short-Context Gains, Long-Context Drift: Negative Results for Token-Conditioned RoPE](https://aclanthology.org/2026.insights-1.13/) (Kazakov, insights 2026)

• Short-Context Gains, Long-Context Drift: Negative Results for Token-Conditioned RoPE (Kazakov, insights 2026)

ACL

• Allan Kazakov. 2026. Short-Context Gains, Long-Context Drift: Negative Results for Token-Conditioned RoPE. In Proceedings of the Seventh Workshop on Insights from Negative Results in NLP, pages 102–109, Budapest, Hungary. Association for Computational Linguistics.