<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bohn-etal-2026-domain">
    <titleInfo>
        <title>Domain-Based Routing in Mixture-of-Experts Models for BabyLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jeremias</namePart>
        <namePart type="family">Bohn</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sebastian</namePart>
        <namePart type="family">Feiertag</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nicolas</namePart>
        <namePart type="family">Raube</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Emil</namePart>
        <namePart type="family">Schwenger</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Georg</namePart>
        <namePart type="family">Groh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second BabyLM Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leshem</namePart>
            <namePart type="family">Choshen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Cotterell</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mustafa</namePart>
            <namePart type="given">Omer</namePart>
            <namePart type="family">Gul</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jaap</namePart>
            <namePart type="family">Jumelet</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tal</namePart>
            <namePart type="family">Linzen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aaron</namePart>
            <namePart type="family">Mueller</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Suchir</namePart>
            <namePart type="family">Salhan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raj</namePart>
            <namePart type="given">Sanjay</namePart>
            <namePart type="family">Shah</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Warstadt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ethan</namePart>
            <namePart type="given">Gotlieb</namePart>
            <namePart type="family">Wilcox</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-475-0</identifier>
    </relatedItem>
    <abstract>Recent language model improvements were mostly achieved by increasing model scale, compute, and in particular training data size. This differs significantly from human language acquisition, which relies on significantly fewer words. The BabyLM challenge tackles this issue by encouraging approaches inspired by cognitive insights to reduce data dependency, thus lowering the barrier for NLP researchers to contribute to their field. We propose a Mixture-of-Experts approach with guided domain-based routing to learn subnetworks specialized on human-interpretable topics, thus making experts focus on domain knowledge and style rather than specific token features. Additionally, we apply a CEFR-based curriculum learning to mimic human language learning. Our model underperforms compared to the 7 times larger baseline model, but scores best on the BabyLM strict-small leaderboard among models with less than 25M parameters. Our code is publicly available at https://github.com/SmallLM-DBR-MoE-Submission-2026/Baby-DBR, and our model can be found at https://huggingface.co/elschr/BabyLM2026-DBR.</abstract>
    <identifier type="citekey">bohn-etal-2026-domain</identifier>
    <location>
        <url>https://aclanthology.org/2026.babylm-main.65/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>843</start>
            <end>858</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Domain-Based Routing in Mixture-of-Experts Models for BabyLMs
%A Bohn, Jeremias
%A Feiertag, Sebastian
%A Raube, Nicolas
%A Schwenger, Emil
%A Groh, Georg
%Y Choshen, Leshem
%Y Cotterell, Ryan
%Y Gul, Mustafa Omer
%Y Jumelet, Jaap
%Y Linzen, Tal
%Y Mueller, Aaron
%Y Salhan, Suchir
%Y Shah, Raj Sanjay
%Y Warstadt, Alex
%Y Wilcox, Ethan Gotlieb
%S Proceedings of the Second BabyLM Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-475-0
%F bohn-etal-2026-domain
%X Recent language model improvements were mostly achieved by increasing model scale, compute, and in particular training data size. This differs significantly from human language acquisition, which relies on significantly fewer words. The BabyLM challenge tackles this issue by encouraging approaches inspired by cognitive insights to reduce data dependency, thus lowering the barrier for NLP researchers to contribute to their field. We propose a Mixture-of-Experts approach with guided domain-based routing to learn subnetworks specialized on human-interpretable topics, thus making experts focus on domain knowledge and style rather than specific token features. Additionally, we apply a CEFR-based curriculum learning to mimic human language learning. Our model underperforms compared to the 7 times larger baseline model, but scores best on the BabyLM strict-small leaderboard among models with less than 25M parameters. Our code is publicly available at https://github.com/SmallLM-DBR-MoE-Submission-2026/Baby-DBR, and our model can be found at https://huggingface.co/elschr/BabyLM2026-DBR.
%U https://aclanthology.org/2026.babylm-main.65/
%P 843-858

Markdown (Informal)

[Domain-Based Routing in Mixture-of-Experts Models for BabyLMs](https://aclanthology.org/2026.babylm-main.65/) (Bohn et al., BabyLM 2026)

• Domain-Based Routing in Mixture-of-Experts Models for BabyLMs (Bohn et al., BabyLM 2026)

ACL

• Jeremias Bohn, Sebastian Feiertag, Nicolas Raube, Emil Schwenger, and Georg Groh. 2026. Domain-Based Routing in Mixture-of-Experts Models for BabyLMs. In Proceedings of the Second BabyLM Workshop, pages 843–858, Budapest, Hungary. Association for Computational Linguistics.