Zichao Wei
Author directory2026
OLMo2 with Hybrid MLM-CLM Training for the BabyLM 2026 Multilingual Challenge
Zichao Wei
Proceedings of the Second BabyLM Workshop
Zichao Wei
Proceedings of the Second BabyLM Workshop
We present our submission to the BabyLM 2026 Multilingual Shared Task. Our system uses OLMo2, a modern decoder-only architecture with grouped-query attention, RoPE, and SwiGLU, trained from scratch on the official BabyLM trilingual corpus (English, Dutch, Chinese). We employ a hybrid training objective following (Charpentier and Samuel, 2024) that combines masked language modeling (87.5%) with causal language modeling (12.5%), enabling the decoder-only model to perform well on both bidirectional and autoregressive evaluation tasks. With a multilingual BPE tokenizer (32K vocabulary) and approximately 79M parameters, our model achieves a multilingual average of 47.12 on the official leaderboard. We release our model and training code.