The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.

As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing systems that retrieve useful documents. We address this gap with SABER-Math, the first fully automated reranking benchmark for mathematical IR, built without expert annotation. Starting from 283K high-school-level problems with solutions, it builds challenging tasks in three steps: (i) LLMs extract concise solution summaries and topics for each problem; (ii) candidates are discovered per query using ontology-based topic similarity and lexical overlap between solution summaries; and (iii) a Swiss-style LLM preference tournament assigns every candidate a graded relevance rating. Over these judged candidate pools, we evaluate retrievers across all major paradigms, from lexical and math-specific systems to modern embedding, late-interaction, and reranking models. Reasoning embedders lead, followed by general-purpose models, both far outperforming classical and math-specific baselines. Yet, all still struggle in symbol-heavy domains such as Algebra and Calculus. Importantly, we show that generic IR benchmarks, such as MTEB, track mathematical performance broadly but are less discriminative among the strongest models, highlighting the need for domain-specific evaluation.

As open-weight large language models (LLMs) achieve ever more impressive performance across a wide range of tasks in English, practitioners aim to adapt these models to different languages. However, such language adaptation is often accompanied by catastrophic forgetting of the base model’s capabilities, severely limiting the usefulness of the resulting model. We address this issue by proposing Branch-and-Merge (BaM), a new adaptation method based on iteratively merging multiple models, fine-tuned on a subset of the available training data. BaM is based on the insight that this yields lower magnitude but higher quality weight changes, reducing forgetting of the source domain while maintaining learning on the target domain. We demonstrate in an extensive empirical study on Bulgarian and German that BaM can significantly reduce forgetting while matching or even improving target domain performance compared to both standard continued pretraining and instruction finetuning across different model architectures.