Recursive Transformers for BabyLM 2026

Serdar Gülbahar; Alexander Fraser; Lukas Edman

Recursive Transformers for BabyLM 2026

Serdar Gülbahar, Alexander Fraser, Lukas Edman

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use ... for bold, ... for italic, ... for underline, <sc>...</sc> for small-caps, <tt>...<tt> for typewriter text, <url>...</url> for URLs, <a href=...> for hyperlinks, and <par/> for paragraph breaks.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

The BabyLM shared task asks whether language models can acquire robust linguistic behavior from developmentally sized corpora. We study a causal transformer recipe for this setting that increases computation per token without adding a full stack of independent layers. Our RecGPT models reuse one transformer block recurrently, factorize the vocabulary maps to keep small models from being dominated by embeddings, and add a training-only next-latent auxiliary loss that encourages hidden states to support local transitions. We submit RecGPT-10M for Strict-Small and RecGPT-100M for Strict. Across the BabyLM 2026 zero-shot and finetuned evaluations, the submitted models improve over GPT-2 baselines and are competitive with GPT-BERT.

Anthology ID:: 2026.babylm-main.51
Volume:: Proceedings of the Second BabyLM Workshop
Month:: October
Year:: 2026
Address:: Budapest, Hungary
Editors:: Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Jaap Jumelet, Tal Linzen, Aaron Mueller, Suchir Salhan, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox
Venues:: BabyLM | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 639–645
Language:
URL:: https://aclanthology.org/2026.babylm-main.51/
DOI:
Bibkey:
Cite (ACL):: Serdar Gülbahar, Alexander Fraser, and Lukas Edman. 2026. Recursive Transformers for BabyLM 2026. In Proceedings of the Second BabyLM Workshop, pages 639–645, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):: Recursive Transformers for BabyLM 2026 (Gülbahar et al., BabyLM 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.babylm-main.51.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{gulbahar-etal-2026-recursive,
    title = "Recursive Transformers for {B}aby{LM} 2026",
    author = {G{\"u}lbahar, Serdar  and
      Fraser, Alexander  and
      Edman, Lukas},
    editor = "Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Salhan, Suchir  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb",
    booktitle = "Proceedings of the Second {B}aby{LM} Workshop",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.babylm-main.51/",
    pages = "639--645",
    ISBN = "979-8-89176-475-0",
    abstract = "The BabyLM shared task asks whether language models can acquire robust linguistic behavior from developmentally sized corpora. We study a causal transformer recipe for this setting that increases computation per token without adding a full stack of independent layers. Our RecGPT models reuse one transformer block recurrently, factorize the vocabulary maps to keep small models from being dominated by embeddings, and add a training-only next-latent auxiliary loss that encourages hidden states to support local transitions. We submit RecGPT-10M for Strict-Small and RecGPT-100M for Strict. Across the BabyLM 2026 zero-shot and finetuned evaluations, the submitted models improve over GPT-2 baselines and are competitive with GPT-BERT."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gulbahar-etal-2026-recursive">
    <titleInfo>
        <title>Recursive Transformers for BabyLM 2026</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Serdar</namePart>
        <namePart type="family">Gülbahar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexander</namePart>
        <namePart type="family">Fraser</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lukas</namePart>
        <namePart type="family">Edman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second BabyLM Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leshem</namePart>
            <namePart type="family">Choshen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mustafa</namePart>
            <namePart type="given">Omer</namePart>
            <namePart type="family">Gul</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jaap</namePart>
            <namePart type="family">Jumelet</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Suchir</namePart>
            <namePart type="family">Salhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raj</namePart>
            <namePart type="given">Sanjay</namePart>
            <namePart type="family">Shah</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Warstadt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ethan</namePart>
            <namePart type="given">Gotlieb</namePart>
            <namePart type="family">Wilcox</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-475-0</identifier>
    </relatedItem>
    <abstract>The BabyLM shared task asks whether language models can acquire robust linguistic behavior from developmentally sized corpora. We study a causal transformer recipe for this setting that increases computation per token without adding a full stack of independent layers. Our RecGPT models reuse one transformer block recurrently, factorize the vocabulary maps to keep small models from being dominated by embeddings, and add a training-only next-latent auxiliary loss that encourages hidden states to support local transitions. We submit RecGPT-10M for Strict-Small and RecGPT-100M for Strict. Across the BabyLM 2026 zero-shot and finetuned evaluations, the submitted models improve over GPT-2 baselines and are competitive with GPT-BERT.</abstract>
    <identifier type="citekey">gulbahar-etal-2026-recursive</identifier>
    <location>
        <url>https://aclanthology.org/2026.babylm-main.51/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>639</start>
            <end>645</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Recursive Transformers for BabyLM 2026
%A Gülbahar, Serdar
%A Fraser, Alexander
%A Edman, Lukas
%Y Choshen, Leshem
%Y Cotterell, Ryan
%Y Gul, Mustafa Omer
%Y Jumelet, Jaap
%Y Linzen, Tal
%Y Mueller, Aaron
%Y Salhan, Suchir
%Y Shah, Raj Sanjay
%Y Warstadt, Alex
%Y Wilcox, Ethan Gotlieb
%S Proceedings of the Second BabyLM Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-475-0
%F gulbahar-etal-2026-recursive
%X The BabyLM shared task asks whether language models can acquire robust linguistic behavior from developmentally sized corpora. We study a causal transformer recipe for this setting that increases computation per token without adding a full stack of independent layers. Our RecGPT models reuse one transformer block recurrently, factorize the vocabulary maps to keep small models from being dominated by embeddings, and add a training-only next-latent auxiliary loss that encourages hidden states to support local transitions. We submit RecGPT-10M for Strict-Small and RecGPT-100M for Strict. Across the BabyLM 2026 zero-shot and finetuned evaluations, the submitted models improve over GPT-2 baselines and are competitive with GPT-BERT.
%U https://aclanthology.org/2026.babylm-main.51/
%P 639-645

Download as File

Markdown (Informal)

[Recursive Transformers for BabyLM 2026](https://aclanthology.org/2026.babylm-main.51/) (Gülbahar et al., BabyLM 2026)

Recursive Transformers for BabyLM 2026 (Gülbahar et al., BabyLM 2026)

ACL

Serdar Gülbahar, Alexander Fraser, and Lukas Edman. 2026. Recursive Transformers for BabyLM 2026. In Proceedings of the Second BabyLM Workshop, pages 639–645, Budapest, Hungary. Association for Computational Linguistics.