Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling

Wiktor Kamzela; Ondřej Dušek; Mateusz Lango

Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling

Wiktor Kamzela, Ondrej Dusek, Mateusz Lango

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use ... for bold, ... for italic, ... for underline, <sc>...</sc> for small-caps, <tt>...<tt> for typewriter text, <url>...</url> for URLs, <a href=...> for hyperlinks, and <par/> for paragraph breaks.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

Neural language model architectures that iterate over their layers have recently shown superior performance on reasoning benchmarks and proved to be a promising method for reducing model size while maintaining performance at scale. However, the effectiveness of such architectures in data-constrained settings remains unexplored. We present a systematic study of layer-sharing transformer architectures trained on the BabyLM corpus (10M tokens), across various iteration strategies and a range of layer and iteration counts. We find that iterative architectures are highly effective under data-constrained setting: a 3-layer iterated model outperforms a standard 24-layer transformer model (ModernBERT-inspired) on SuperGLUE benchmark.

Anthology ID:: 2026.babylm-main.54
Volume:: Proceedings of the Second BabyLM Workshop
Month:: October
Year:: 2026
Address:: Budapest, Hungary
Editors:: Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Jaap Jumelet, Tal Linzen, Aaron Mueller, Suchir Salhan, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox
Venues:: BabyLM | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 678–688
Language:
URL:: https://aclanthology.org/2026.babylm-main.54/
DOI:
Bibkey:
Cite (ACL):: Wiktor Kamzela, Ondrej Dusek, and Mateusz Lango. 2026. Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling. In Proceedings of the Second BabyLM Workshop, pages 678–688, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):: Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling (Kamzela et al., BabyLM 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.babylm-main.54.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{kamzela-etal-2026-scaling,
    title = "Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling",
    author = "Kamzela, Wiktor  and
      Dusek, Ondrej  and
      Lango, Mateusz",
    editor = "Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Salhan, Suchir  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb",
    booktitle = "Proceedings of the Second {B}aby{LM} Workshop",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.babylm-main.54/",
    pages = "678--688",
    ISBN = "979-8-89176-475-0",
    abstract = "Neural language model architectures that iterate over their layers have recently shown superior performance on reasoning benchmarks and proved to be a promising method for reducing model size while maintaining performance at scale. However, the effectiveness of such architectures in data-constrained settings remains unexplored. We present a systematic study of layer-sharing transformer architectures trained on the BabyLM corpus (10M tokens), across various iteration strategies and a range of layer and iteration counts. We find that iterative architectures are highly effective under data-constrained setting: a 3-layer iterated model outperforms a standard 24-layer transformer model (ModernBERT-inspired) on SuperGLUE benchmark."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kamzela-etal-2026-scaling">
    <titleInfo>
        <title>Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Wiktor</namePart>
        <namePart type="family">Kamzela</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ondrej</namePart>
        <namePart type="family">Dusek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mateusz</namePart>
        <namePart type="family">Lango</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second BabyLM Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leshem</namePart>
            <namePart type="family">Choshen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mustafa</namePart>
            <namePart type="given">Omer</namePart>
            <namePart type="family">Gul</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jaap</namePart>
            <namePart type="family">Jumelet</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Suchir</namePart>
            <namePart type="family">Salhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raj</namePart>
            <namePart type="given">Sanjay</namePart>
            <namePart type="family">Shah</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Warstadt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ethan</namePart>
            <namePart type="given">Gotlieb</namePart>
            <namePart type="family">Wilcox</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-475-0</identifier>
    </relatedItem>
    <abstract>Neural language model architectures that iterate over their layers have recently shown superior performance on reasoning benchmarks and proved to be a promising method for reducing model size while maintaining performance at scale. However, the effectiveness of such architectures in data-constrained settings remains unexplored. We present a systematic study of layer-sharing transformer architectures trained on the BabyLM corpus (10M tokens), across various iteration strategies and a range of layer and iteration counts. We find that iterative architectures are highly effective under data-constrained setting: a 3-layer iterated model outperforms a standard 24-layer transformer model (ModernBERT-inspired) on SuperGLUE benchmark.</abstract>
    <identifier type="citekey">kamzela-etal-2026-scaling</identifier>
    <location>
        <url>https://aclanthology.org/2026.babylm-main.54/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>678</start>
            <end>688</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling
%A Kamzela, Wiktor
%A Dusek, Ondrej
%A Lango, Mateusz
%Y Choshen, Leshem
%Y Cotterell, Ryan
%Y Gul, Mustafa Omer
%Y Jumelet, Jaap
%Y Linzen, Tal
%Y Mueller, Aaron
%Y Salhan, Suchir
%Y Shah, Raj Sanjay
%Y Warstadt, Alex
%Y Wilcox, Ethan Gotlieb
%S Proceedings of the Second BabyLM Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-475-0
%F kamzela-etal-2026-scaling
%X Neural language model architectures that iterate over their layers have recently shown superior performance on reasoning benchmarks and proved to be a promising method for reducing model size while maintaining performance at scale. However, the effectiveness of such architectures in data-constrained settings remains unexplored. We present a systematic study of layer-sharing transformer architectures trained on the BabyLM corpus (10M tokens), across various iteration strategies and a range of layer and iteration counts. We find that iterative architectures are highly effective under data-constrained setting: a 3-layer iterated model outperforms a standard 24-layer transformer model (ModernBERT-inspired) on SuperGLUE benchmark.
%U https://aclanthology.org/2026.babylm-main.54/
%P 678-688

Download as File

Markdown (Informal)

[Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling](https://aclanthology.org/2026.babylm-main.54/) (Kamzela et al., BabyLM 2026)

Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling (Kamzela et al., BabyLM 2026)

ACL

Wiktor Kamzela, Ondrej Dusek, and Mateusz Lango. 2026. Scaling Compute Instead of Parameters: Exploring Iterative Transformers for Data-Efficient Language Modeling. In Proceedings of the Second BabyLM Workshop, pages 678–688, Budapest, Hungary. Association for Computational Linguistics.