Nicolas Raube
Author directory2026
Domain-Based Routing in Mixture-of-Experts Models for BabyLMs
Jeremias Bohn | Sebastian Feiertag | Nicolas Raube | Emil Schwenger | Georg Groh
Proceedings of the Second BabyLM Workshop
Jeremias Bohn | Sebastian Feiertag | Nicolas Raube | Emil Schwenger | Georg Groh
Proceedings of the Second BabyLM Workshop
Recent language model improvements were mostly achieved by increasing model scale, compute, and in particular training data size. This differs significantly from human language acquisition, which relies on significantly fewer words. The BabyLM challenge tackles this issue by encouraging approaches inspired by cognitive insights to reduce data dependency, thus lowering the barrier for NLP researchers to contribute to their field. We propose a Mixture-of-Experts approach with guided domain-based routing to learn subnetworks specialized on human-interpretable topics, thus making experts focus on domain knowledge and style rather than specific token features. Additionally, we apply a CEFR-based curriculum learning to mimic human language learning. Our model underperforms compared to the 7 times larger baseline model, but scores best on the BabyLM strict-small leaderboard among models with less than 25M parameters. Our code is publicly available at https://github.com/SmallLM-DBR-MoE-Submission-2026/Baby-DBR, and our model can be found at https://huggingface.co/elschr/BabyLM2026-DBR.