Naoki Mori

Author directory

2026

Under a fixed multilingual pretraining budget, we study how language allocation, data design, and model configuration affect English–Dutch–Chinese pretraining. First, we trained three fixed language mixtures with three matched seeds. No pairwise difference in the mean of seven zero-shot benchmarks used for model selection was significant after Holm correction. Language-specific scores nevertheless differed: the 50/25/25 mixture scored 2.487 points higher than 27.5/27.5/45 in English, whereas 27.5/27.5/45 scored 2.151 points higher than 40/40/20 in Chinese, after correction within each language. In a 25M-word proxy experiment, an inverse-score controller changed language exposure but did not improve the aggregate score. Second, across four matched seeds, the targeted-augmentation condition scored higher on ZS16 than the real-text baseline (57.37 versus 56.57). The largest language-specific difference was observed for Chinese (+1.47). Both the ZS16 and Chinese differences remained significant after Holm correction over the four reported tests. Across three matched seeds, multi-token prediction showed no detectable improvement in ZS16. In single-run comparisons across parameter counts, the 213M-parameter model had the highest observed FT22 score (41.25). We treat all parameter-count comparisons and the remaining tokenizer and data ablations as descriptive because each used a single run. The 213M-parameter model obtained a leaderboard-reported Multilingual Average of 50.68.