Melike Uysal

Author directory

2026

The development of Large Language Models has primarily focused on English-centric or massively multilingual paradigms. As a result, the dynamics of mixing just two languages from the very beginning of training remain underexplored, especially for smaller models with limited capacity. In this study, we conduct a controlled set of experiments to evaluate the impact of bilingual corpus during from-scratch pretraining. Using a capacity-constrained Gemma-4-derived 181M-parameter architecture trained on a 10 billion token budget, we compare a purely Turkish baseline against mixtures containing 10%, 20%, or 30% of an auxiliary language. We test four languages to isolate different linguistic relationships: English (a global bridge language), Azerbaijani (closely related sibling), Finnish (morphological peer), and French (lexical pivot). Our findings complicate the curse of multilinguality, the tendency for per-language performance to drop as more languages are added. We show that adding any secondary language hurts fluency on structurally uniform text. However, co-training with a typological sibling, Azerbaijani, cuts perplexity on creative and literary text by up to 45% and 51% respectively relative to the monolingual baseline. At higher mixture ratios, Azerbaijani outperforms other languages including morphological and lexical alternatives.