Chain-of-Thought in a Low-Resource Language: A Factorial Study of Exemplar and Reasoning Language Effects in Urdu

Ateeb Ahmed


Abstract
Chain-of-thought (CoT) prompting has been shown theoretically to extend the computational capacity of transformers by providing intermediate reasoning steps (Feng et al., 2023; Amiri et al., 2025). Whether these benefits carry over to low-resource languages is not well understood, and Urdu in particular has received very little attention in this context. We investigate two prompt factors (the language of few-shot exemplars and the language of the reasoning chain) using a 2 × 2 factorial design on MGSM Urdu and Math-500 Urdu, evaluated with Gemini 2.5 Flash. We find a significant interaction between exemplar and CoT language (item-level logistic regression, z = 6.9, p < 0.001): switching to Urdu CoT with English exemplars causes a 34.8 percentage point accuracy drop on MGSM, while the same switch with Urdu exemplars results in only a 1.6 point drop. We also find that Urdu CoT produces longer token sequences than English CoT (139 vs. 101 tokens on average) but not correspondingly more discrete reasoning steps: a script-aware, manually validated step count shows Urdu CoT yields slightly fewer discrete steps (2.5 vs. 3.0 on average), with the reduction concentrated in the mismatchedexemplar condition, indicating that added verbosity does not translate into added reasoning structure. Chain length correlates positively with accuracy (r = 0.42, p < 0.001), consistent with the theoretical framework. The main takeaway is that exemplar language is a critical moderator of CoT effectiveness in Urdu.
Anthology ID:
2026.oracle-main.5
Volume:
The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE)
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, Marco Valentino, Alexandra Birch, Fabio Massimo Zanzotto
Venues:
ORACLE | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
75–83
Language:
URL:
https://aclanthology.org/2026.oracle-main.5/
DOI:
Bibkey:
Cite (ACL):
Ateeb Ahmed. 2026. Chain-of-Thought in a Low-Resource Language: A Factorial Study of Exemplar and Reasoning Language Effects in Urdu. In The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE), pages 75–83, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Chain-of-Thought in a Low-Resource Language: A Factorial Study of Exemplar and Reasoning Language Effects in Urdu (Ahmed, ORACLE 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.oracle-main.5.pdf