Kai-hao Yang
Author directory2026
Optimizing Multilingual BabyLM: The Roles of Tokenization, Data Packing, and Curriculum Learning
Fuming Yang | Kai-hao Yang | Hsuan Lien | Lukas Edman | Alexander Fraser
Proceedings of the Second BabyLM Workshop
Fuming Yang | Kai-hao Yang | Hsuan Lien | Lukas Edman | Alexander Fraser
Proceedings of the Second BabyLM Workshop
Training multilingual language models under strict data constraints remains a challenging problem. This paper presents our submission to the BabyLM 2026 Multilingual Track, focusing on English, Dutch, and Chinese. Three key challenges arise under the BabyLM constraints: efficiently sharing limited model capacity, maximizing sequence utilization, and allocating parameters efficiently under a fixed vocabulary budget. To address these challenges, we propose a three-stage curriculum learning strategy that dynamically adjusts language mixing ratios throughout training. Furthermore, we investigate data packing strategies, comparing naive chunking with document wrapping to maximize the learning signal within a strict 100M-token budget. Finally, we design a custom BPE tokenizer with an asymmetric sampling strategy. By reducing the size of the embedding matrix, the saved parameters are reallocated to the Transformer backbone, enabling highly efficient GPT-2 and BERT models. Experimental results show that combining our curriculum strategy with document packing consistently improves GPT-2 training convergence and achieves highly competitive multilingual performance.