<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ploner-akbik-2026-randomly">
    <titleInfo>
        <title>Can Randomly Permuting the LM Decoding Order Boost Data-Efficiency?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Max</namePart>
        <namePart type="family">Ploner</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alan</namePart>
        <namePart type="family">Akbik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second BabyLM Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leshem</namePart>
            <namePart type="family">Choshen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mustafa</namePart>
            <namePart type="given">Omer</namePart>
            <namePart type="family">Gul</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jaap</namePart>
            <namePart type="family">Jumelet</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Suchir</namePart>
            <namePart type="family">Salhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raj</namePart>
            <namePart type="given">Sanjay</namePart>
            <namePart type="family">Shah</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Warstadt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ethan</namePart>
            <namePart type="given">Gotlieb</namePart>
            <namePart type="family">Wilcox</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-475-0</identifier>
    </relatedItem>
    <abstract>Causal language models, typically trained with next-token prediction, condition each token’s probability on all preceding tokens. When pretraining is limited by the amount of available training data (rather than compute), repeating training over the same corpus yields a largely unchanged learning signal across epochs. In contrast, permutation language models (PLMs) predict tokens in a random order, conditioning each prediction on a bidirectional set of tokens and thereby increasing the diversity of contextual evidence a token can receive. We hypothesize that PLMs (1) profit more from longer training on repeated data, (2) outperform classical CLMs in data-limited settings where multiple epochs are allowed, and (3) excel on tasks where bidirectional context is useful.\par To test these hypotheses, we train multiple models on the BabyLM strict-small pretraining data consisting of only 10M words and evaluate how decoding order and bidirectional conditioning affect model performance under tight training-data budgets. We find that while PLMs profit from longer training, they do not generally outperform CLMs except on tasks where use of bidirectional context is required.</abstract>
    <identifier type="citekey">ploner-akbik-2026-randomly</identifier>
    <location>
        <url>https://aclanthology.org/2026.babylm-main.24/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>299</start>
            <end>308</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can Randomly Permuting the LM Decoding Order Boost Data-Efficiency?
%A Ploner, Max
%A Akbik, Alan
%Y Choshen, Leshem
%Y Cotterell, Ryan
%Y Gul, Mustafa Omer
%Y Jumelet, Jaap
%Y Linzen, Tal
%Y Mueller, Aaron
%Y Salhan, Suchir
%Y Shah, Raj Sanjay
%Y Warstadt, Alex
%Y Wilcox, Ethan Gotlieb
%S Proceedings of the Second BabyLM Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-475-0
%F ploner-akbik-2026-randomly
%X Causal language models, typically trained with next-token prediction, condition each token’s probability on all preceding tokens. When pretraining is limited by the amount of available training data (rather than compute), repeating training over the same corpus yields a largely unchanged learning signal across epochs. In contrast, permutation language models (PLMs) predict tokens in a random order, conditioning each prediction on a bidirectional set of tokens and thereby increasing the diversity of contextual evidence a token can receive. We hypothesize that PLMs (1) profit more from longer training on repeated data, (2) outperform classical CLMs in data-limited settings where multiple epochs are allowed, and (3) excel on tasks where bidirectional context is useful.\par To test these hypotheses, we train multiple models on the BabyLM strict-small pretraining data consisting of only 10M words and evaluate how decoding order and bidirectional conditioning affect model performance under tight training-data budgets. We find that while PLMs profit from longer training, they do not generally outperform CLMs except on tasks where use of bidirectional context is required.
%U https://aclanthology.org/2026.babylm-main.24/
%P 299-308

Markdown (Informal)

[Can Randomly Permuting the LM Decoding Order Boost Data-Efficiency?](https://aclanthology.org/2026.babylm-main.24/) (Ploner & Akbik, BabyLM 2026)

• Can Randomly Permuting the LM Decoding Order Boost Data-Efficiency? (Ploner & Akbik, BabyLM 2026)

ACL

• Max Ploner and Alan Akbik. 2026. Can Randomly Permuting the LM Decoding Order Boost Data-Efficiency?. In Proceedings of the Second BabyLM Workshop, pages 299–308, Budapest, Hungary. Association for Computational Linguistics.