Recent advances in large language models (LLMs) have led to strong reasoning capabilities; however, evaluating such models in lowresource languages remains challenging due to the lack of standardized reasoning benchmarks. Urdu reasoning evaluation, in particular, is limited by machine translation sensitivity and an emphasis on general language tasks rather than reasoning benchmarks. We propose a contextually ensembled translation framework with human-in-the-loop validation that leverages multiple translation systems to develop Urdu reasoning benchmarks while preserving contextual and structural integrity. Using this framework, we translate widely adopted benchmarks, including MGSM, MATH-500, CommonSenseQA (CSQA), and OpenBookQA (OBQA), into Urdu, collectively referred to as UrduBench, and evaluate reasoning-oriented and instruction-tuned LLMs across multiple prompting strategies. Our analysis examines four datasets, five task difficulty levels, diverse model architectures, model scaling settings, and language consistency. We find that multistep and symbolic reasoning tasks pose significant challenges in Urdu, and that stable language alignment is a critical prerequisite for robust reasoning. Overall, our work establishes a scalable methodology for standardized Urdu reasoning evaluation and provides insights into multilingual reasoning failures, with applicability to other low-resource languages. The code and datasets will be publicly released.

Developing a high-performing large language models (LLMs) for low-resource languages such as Urdu, present several challenges. These challenges include the scarcity of high-quality datasets, multilingual inconsistencies, and safety concerns. Existing multilingual LLMs often address these issues by translating large volumes of available data. However, such translations often lack quality and cultural nuance while also incurring significant costs for data curation and training. To address these issues, we propose Alif-1.0-8B-Instruct, a multilingual Urdu-English model, that tackles these challenges with a unique approach. We train the model on a high-quality, multilingual synthetic dataset (Urdu-Instruct), developed using a modified self-instruct technique. By using unique prompts and seed values for each task along with a global task pool, this dataset incorporates Urdu-native chain-of-thought based reasoning, bilingual translation, cultural relevance, and ethical safety alignments. This technique significantly enhances the comprehension of Alif-1.0-8B-Instruct model for Urdu-specific tasks. As a result, Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks. It also outperformed leading multilingual LLMs, including Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B, all within a training budget of under $100. Our results demonstrate that high-performance and low-resource language LLMs can be developed efficiently and culturally aligned using our modified self-instruct approach.