As large language models saturate existing benchmarks, language- and culture-specific tasks remain a productive source of problems that are challenging for current models yet solvable by humans. We present SQARB (Sophisticated Question Answering and Reasoning in Belarusian), the first dedicated, natively authored question answering resource for Belarusian, a low-resource East Slavic language. SQARB contains 500 short-answer questions manually written by domain experts, based on doctoral dissertations defended in Belarus between 1994 and 2023, covering five topics: the Belarusian language, literature, folklore, art, and history. The questions are designed to be culture-specific, hard to retrieve online, evergreen, and to require a mix of factual knowledge and reasoning. The dataset is split into an easier development set and a harder test set, both filtered adversarially against locally deployed models. We evaluate near-frontier models without tool use on SQARB and find a large gap between proprietary and open-weight models, with the best-performing model (Gemini 3.1 Pro) achieving 42.3% accuracy on the test set. Models consistently exhibit overconfidence and perform relatively better on questions requiring plausible reasoning, rather than pure factual recall. We release SQARB under gated access with a canary string.1

We examine how the capabilities of large language models (LLMs) have evolved on eight Belarusian language tasks contributed in 2023 to OpenAI’s Evals framework. We evaluate state-of-the-art models both on the original development sets and newly created test sets. Results demonstrate significant but non-uniform progress over this period: some tasks are almost saturated, while others show minor improvement beyond trivial baselines. Error analysis shows that certain challenges haven’t yet been addressed, e.g. misidentification of non-words as legitimate vocabulary, or conversion from modern to classical orthography. We release the datasets and the generated completions (https://doi.org/10.5281/zenodo.18163825).

In the epoch of multilingual large language models (LLMs), it is still challenging to evaluate the models’ understanding of lower-resourced languages, which motivates further development of expert-crafted natural language understanding benchmarks. We introduce BelarusianGLUE — a natural language understanding benchmark for Belarusian, an East Slavic language, with ≈15K instances in five tasks: sentiment analysis, linguistic acceptability, word in context, Winograd schema challenge, textual entailment. A systematic evaluation of BERT models and LLMs against this novel benchmark reveals that both types of models approach human-level performance on easier tasks, such as sentiment analysis, but there is a significant gap in performance between machine and human on a harder task — Winograd schema challenge. We find the optimal choice of model type to be task-specific: e.g. BERT models underperform on textual entailment task but are competitive for linguistic acceptability. We release the datasets (https://hf.co/datasets/maaxap/BelarusianGLUE) and evaluation code (https://github.com/maaxap/BelarusianGLUE).