As LLMs advance, popular commonsense and NLU benchmarks have become saturated, yet multilingual evaluation still lags. Existing benchmarks cover only a small set of languages, and translation based multilingual benchmarks can introduce bias. We introduce HellaSwagUltra, a multilingual commonsense and NLU benchmark with substantial local cultural knowledge. It is built with an automated, continuously extensible pipeline. Instead of explicitly prompting for reasoning, each example implicitly embeds two commonsense or local knowledge facts in the context, and answer options contain subtle cues about whether those facts are violated, requiring models to pick the most plausible continuation. All test items are fully reviewed and corrected by language experts, with ongoing updates. Even the strongest proprietary model (e.g., Gemini-2.5-Pro) reaches only 62.5% accuracy, while GPT-4o and leading open-source models hover around 40%–50%, demonstrating that multilingual commonsense reasoning remains an open challenge.

Multilingual large language models (LLMs) are advancing rapidly, with new models frequently claiming support for an increasing number of languages. However, existing evaluation datasets are limited and lack cross-lingual alignment, leaving assessments of multilingual capabilities fragmented in both language and skill coverage. To address this, we introduce MuBench, a benchmark covering 61 languages with 3.9M samples and evaluating a broad range of capabilities. We evaluate several state-of-the-art multilingual LLMs and find notable gaps between claimed and actual language coverage, particularly a persistent performance disparity between English and low-resource languages. Leveraging MuBench’s alignment, we propose Multilingual Consistency (MLC) as a complementary metric to accuracy for analyzing performance bottlenecks and guiding model improvement. MuBench provides flexible evaluation formats, including mixed-language testing. Experimental results show that increasing model size does not improve its ability to handle mixed-language contexts. We recruited human experts to evaluate translation quality and cultural sensitivity for 34k samples across 17 languages, and combined these assessments with an LLM-as-a-Judge approach to ensure overall data quality in low resource languages.

Multilingual LLMs increasingly claim broad language coverage, yet their performance still lags far behind English in many languages. Because low resource data is scarce and noisy, improving these languages directly is difficult, making cross lingual transfer from English a natural alternative. Parallel corpora are central to such transfer, but it remains unclear which kinds of parallel data matter most. We propose ParaRater, a corpus selection method that identifies high value English text to translate into target languages, building parallel corpora that most efficiently improve target language performance. ParaRater uses meta learning to tie selection directly to downstream accuracy on native target language data, and applies a two stage filter to find samples that only help when both language versions are used in training, capturing truly impactful parallel data. Experiments across Chinese, German, French, Japanese, Thai, Arabic, Tagalog, and Russian, together with qualitative analyses, support these findings.