Areej Mehboob
Author directory2026
UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop
Muhammad Ali Shafique | Areej Mehboob | Layba Fiaz | Muhammad Usman Qadeer | Hamza Farooq
The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
Muhammad Ali Shafique | Areej Mehboob | Layba Fiaz | Muhammad Usman Qadeer | Hamza Farooq
The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
Recent advances in large language models (LLMs) have led to strong reasoning capabilities; however, evaluating such models in lowresource languages remains challenging due to the lack of standardized reasoning benchmarks. Urdu reasoning evaluation, in particular, is limited by machine translation sensitivity and an emphasis on general language tasks rather than reasoning benchmarks. We propose a contextually ensembled translation framework with human-in-the-loop validation that leverages multiple translation systems to develop Urdu reasoning benchmarks while preserving contextual and structural integrity. Using this framework, we translate widely adopted benchmarks, including MGSM, MATH-500, CommonSenseQA (CSQA), and OpenBookQA (OBQA), into Urdu, collectively referred to as UrduBench, and evaluate reasoning-oriented and instruction-tuned LLMs across multiple prompting strategies. Our analysis examines four datasets, five task difficulty levels, diverse model architectures, model scaling settings, and language consistency. We find that multistep and symbolic reasoning tasks pose significant challenges in Urdu, and that stable language alignment is a critical prerequisite for robust reasoning. Overall, our work establishes a scalable methodology for standardized Urdu reasoning evaluation and provides insights into multilingual reasoning failures, with applicability to other low-resource languages. The code and datasets will be publicly released.