Melike Uysal
Author directory2026
Impact of Bilingual Corpus on Pretraining Large Language Models: An Analysis for Turkish
Elif Özge Yılmaz | Melike Uysal | Cagri Toraman
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
Elif Özge Yılmaz | Melike Uysal | Cagri Toraman
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
The development of Large Language Models has primarily focused on English-centric or massively multilingual paradigms. As a result, the dynamics of mixing just two languages from the very beginning of training remain underexplored, especially for smaller models with limited capacity. In this study, we conduct a controlled set of experiments to evaluate the impact of bilingual corpus during from-scratch pretraining. Using a capacity-constrained Gemma-4-derived 181M-parameter architecture trained on a 10 billion token budget, we compare a purely Turkish baseline against mixtures containing 10%, 20%, or 30% of an auxiliary language. We test four languages to isolate different linguistic relationships: English (a global bridge language), Azerbaijani (closely related sibling), Finnish (morphological peer), and French (lexical pivot). Our findings complicate the curse of multilinguality, the tendency for per-language performance to drop as more languages are added. We show that adding any secondary language hurts fluency on structurally uniform text. However, co-training with a typological sibling, Azerbaijani, cuts perplexity on creative and literary text by up to 45% and 51% respectively relative to the monolingual baseline. At higher mixture ratios, Azerbaijani outperforms other languages including morphological and lexical alternatives.