<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wang-2026-input">
    <titleInfo>
        <title>What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yupei</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second BabyLM Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leshem</namePart>
            <namePart type="family">Choshen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mustafa</namePart>
            <namePart type="given">Omer</namePart>
            <namePart type="family">Gul</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jaap</namePart>
            <namePart type="family">Jumelet</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Suchir</namePart>
            <namePart type="family">Salhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raj</namePart>
            <namePart type="given">Sanjay</namePart>
            <namePart type="family">Shah</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Warstadt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ethan</namePart>
            <namePart type="given">Gotlieb</namePart>
            <namePart type="family">Wilcox</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-475-0</identifier>
    </relatedItem>
    <abstract>Learned weighting of earlier layers, as used in ELC-BERT (Charpentier and Samuel, 2023), is a promising direction for sample-efficient language modeling. Attention Residuals (AttnRes; Kimi Team et al., 2026) extend this idea by computing input-dependent weights separately for each token, with reported gains in large-scale autoregressive models. We present a controlled study of whether such input-dependent depth routing helps in the data-limited regime where static routing first proved useful. Across the 10M- and 100M-word BabyLM budgets and three paired seeds, AttnRes consistently lowers test negative log-likelihood (NLL) compared to a residual baseline. It reaches the same development-set (dev) NLL after seeing fewer training words. In the 10M L32 control, AttnRes also lowers test NLL relative to static routing in all three seeds. Behavioral differences between AttnRes and the baseline are small and mixed. The BLiMP (Warstadt et al., 2020) advantage at equal training exposure disappears after matching on dev NLL. The trained 10M-word AttnRes models make causal use of routes from nonrecent depth blocks, but we find no clear evidence that categories with greater dependence on these sources achieve larger BLiMP gains over the baseline.</abstract>
    <identifier type="citekey">wang-2026-input</identifier>
    <location>
        <url>https://aclanthology.org/2026.babylm-main.68/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>892</start>
            <end>903</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling?
%A Wang, Yupei
%Y Choshen, Leshem
%Y Cotterell, Ryan
%Y Gul, Mustafa Omer
%Y Jumelet, Jaap
%Y Linzen, Tal
%Y Mueller, Aaron
%Y Salhan, Suchir
%Y Shah, Raj Sanjay
%Y Warstadt, Alex
%Y Wilcox, Ethan Gotlieb
%S Proceedings of the Second BabyLM Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-475-0
%F wang-2026-input
%X Learned weighting of earlier layers, as used in ELC-BERT (Charpentier and Samuel, 2023), is a promising direction for sample-efficient language modeling. Attention Residuals (AttnRes; Kimi Team et al., 2026) extend this idea by computing input-dependent weights separately for each token, with reported gains in large-scale autoregressive models. We present a controlled study of whether such input-dependent depth routing helps in the data-limited regime where static routing first proved useful. Across the 10M- and 100M-word BabyLM budgets and three paired seeds, AttnRes consistently lowers test negative log-likelihood (NLL) compared to a residual baseline. It reaches the same development-set (dev) NLL after seeing fewer training words. In the 10M L32 control, AttnRes also lowers test NLL relative to static routing in all three seeds. Behavioral differences between AttnRes and the baseline are small and mixed. The BLiMP (Warstadt et al., 2020) advantage at equal training exposure disappears after matching on dev NLL. The trained 10M-word AttnRes models make causal use of routes from nonrecent depth blocks, but we find no clear evidence that categories with greater dependence on these sources achieve larger BLiMP gains over the baseline.
%U https://aclanthology.org/2026.babylm-main.68/
%P 892-903

Markdown (Informal)

[What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling?](https://aclanthology.org/2026.babylm-main.68/) (Wang, BabyLM 2026)

• What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling? (Wang, BabyLM 2026)

ACL

• Yupei Wang. 2026. What Does Input-Dependent Depth Routing Buy in Data-Limited Language Modeling?. In Proceedings of the Second BabyLM Workshop, pages 892–903, Budapest, Hungary. Association for Computational Linguistics.