<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="haller-etal-2026-matters">
    <titleInfo>
        <title>What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Patrick</namePart>
        <namePart type="family">Haller</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jonas</namePart>
        <namePart type="family">Golde</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alan</namePart>
        <namePart type="family">Akbik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Linearization has emerged as a strategy for developing efficient language models (LMs). Starting from an existing Transformer-based LM, linearization replaces the attention component with computationally efficient subquadratic token mixers. However, as an increasing number of mixers are proposed, it remains unclear which inductive biases are best suited to inherit the original Transformer’s capabilities. Furthermore, it is unknown how linearization is affected by parameter and token budget scaling. To address these questions, we propose a unified setup to compare seven representative architectures, including xLSTM, GLA, and Gated DeltaNet. Our findings reveal that performance hierarchies remain stable from 140M to 1.7B parameters, with error-correcting update rules demonstrating superior scaling exponents. We show that performance gaps are established early and persist through asymptotic maturity at 10B tokens, suggesting that state resolution is a more fundamental bottleneck than the distillation budget. Finally, while most models adapt to instruction tuning, only gated delta-rule formulations maintain the precision necessary for long-context retrieval, whereas additive models suffer from irreversible state saturation. Finally, we show that a cheap mixer-only probe predicts these downstream outcomes before a full run and, used as a diagnostic, attributes the weakest student’s scaling deficit to a fixable normalization choice rather than its update rule. These results suggest that for successful linearization, architectural inductive biases remain the primary constraint that cannot be overcome by simply scaling training compute.</abstract>
    <identifier type="citekey">haller-etal-2026-matters</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.2648/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>60257</start>
            <end>60274</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation
%A Haller, Patrick
%A Golde, Jonas
%A Akbik, Alan
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F haller-etal-2026-matters
%X Linearization has emerged as a strategy for developing efficient language models (LMs). Starting from an existing Transformer-based LM, linearization replaces the attention component with computationally efficient subquadratic token mixers. However, as an increasing number of mixers are proposed, it remains unclear which inductive biases are best suited to inherit the original Transformer’s capabilities. Furthermore, it is unknown how linearization is affected by parameter and token budget scaling. To address these questions, we propose a unified setup to compare seven representative architectures, including xLSTM, GLA, and Gated DeltaNet. Our findings reveal that performance hierarchies remain stable from 140M to 1.7B parameters, with error-correcting update rules demonstrating superior scaling exponents. We show that performance gaps are established early and persist through asymptotic maturity at 10B tokens, suggesting that state resolution is a more fundamental bottleneck than the distillation budget. Finally, while most models adapt to instruction tuning, only gated delta-rule formulations maintain the precision necessary for long-context retrieval, whereas additive models suffer from irreversible state saturation. Finally, we show that a cheap mixer-only probe predicts these downstream outcomes before a full run and, used as a diagnostic, attributes the weakest student’s scaling deficit to a fixable normalization choice rather than its update rule. These results suggest that for successful linearization, architectural inductive biases remain the primary constraint that cannot be overcome by simply scaling training compute.
%U https://aclanthology.org/2026.emnlp-main.2648/
%P 60257-60274

Markdown (Informal)

[What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation](https://aclanthology.org/2026.emnlp-main.2648/) (Haller et al., EMNLP 2026)

• What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation (Haller et al., EMNLP 2026)

ACL

• Patrick Haller, Jonas Golde, and Alan Akbik. 2026. What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 60257–60274, Budapest, Hungary. Association for Computational Linguistics.