Kazakhstan is transitioning the Kazakh language from Cyrillic to a Latin alphabet, with four official Latin proposals issued since 2017. How well do current large language models, trained almost entirely on Cyrillic Kazakh, handle this script change? We evaluate 13 models spanning general-purpose open-weight, Kazakh-adapted, and proprietary systems on three Kazakh benchmarks (KazMMLU, KazCulture, GSM8k-Kazakh) rewritten in all four Latin variants. The mean accuracy drop from Cyrillic to Latin is 16.8 points, ranging from 1.4 (GPT-5) to 30.4 (Sherkala-8B). Contrary to the natural tokenization-explosion hypothesis, fertility analysis shows that Latin Kazakh requires no more tokens per word than Cyrillic for most tokenizers. Subword fragmentation therefore contributes to the drop but does not primarily explain it. Kazakh-adapted models drop hardest, suggesting that Cyrillic-only adaptation entrenches script-specific routes rather than building script-agnostic competence. Prompt-level mitigations (hint, mapping) recover only 1 point on average. On the other hand, LoRA fine-tuning recovers up to +20 points, but only for sufficiently large reasoning-capable bases. Generative AI will play a key role in the script transition. Models serving Kazakh users must reliably process Latin script, which our findings show cannot be achieved at inference time. Addressing this gap requires incorporating Latin Kazakh data during pretraining or large-scale continued pretraining, rather than relying solely on prompt or adapter-level solutions.

We conducted a systematic evaluation of seven large language models (LLMs) on tasks in Kazakh, a Turkic language spoken by approximately 13 million native speakers in Kazakhstan and abroad. We used six datasets corresponding to different tasks – questions answering, causal reasoning, middle school math problems, machine translation, and spelling correction. Three of the datasets were prepared for this study. As expected, the quality of the LLMs on the Kazakh tasks is lower than on the parallel English tasks. GPT-4 shows the best results, followed by Gemini and . In general, LLMs perform better on classification tasks and struggle with generative tasks. Our results provide valuable insights into the applicability of currently available LLMs for Kazakh. We made the data collected for this study publicly available: https://github.com/akylbekmaxutov/LLM-eval-using-Kazakh.