This paper introduces PSMT-eval, a new dataset for machine translation (MT) evaluation in Punjabi Shahmukhi, a low-resource language spoken by over 80 million people. We constructed an English–Punjabi Shahmukhi parallel corpus through pivot-based human translation, where the Urdu side of existing English–Urdu datasets was manually translated into Punjabi Shahmukhi. Four widely used English–Urdu datasets were selected: tatoeba-test.eng-urd, FLORES101, IN22, and MKB, with total count of 5182 sentences. To assess the dataset, we evaluated translations generated by state-of-the-art large language models (LLMs), including GPT-5, GPT-OSS-120B, Claude 4 Sonnet, LLaMA-3.3 70B Instruct, and Qwen-3 32B. Results show that only Claude 4 Sonnet achieved competitive performance, with a maximum SacreBLEU score of 17.51 on IN22. GPT-5 followed with a maximum SacreBLEU score of 10, while other models performed poorly. Further analysis highlights major challenges in Punjabi Shahmukhi MT. To the best of our knowledge, this is the first benchmark dataset specifically for MT evaluation in Punjabi Shahmukhi. PSMT-eval is a valuable resource for advancing low-resource MT research and is publicly available to support future work.

Recent advancements in Neural Machine Translation (NMT) systems have significantly improved model performance on various translation benchmarks. However, these systems still face numerous challenges when translating low-resource languages such as Urdu. In this work, we highlight the specific issues faced by machine translation systems when translating Urdu language. We first conduct a comprehensive evaluation of English to Urdu Machine Translation with four diverse models: GPT-3.5 (a large language model), opus-mt-en-ur (a bilingual translation model), NLLB (a model trained for translating 200 languages), and IndicTrans2 (a specialized model for translating low-resource Indic languages). The results demonstrate that IndicTrans2 significantly outperforms other models in Urdu Machine Translation. To understand the differences in the performance of these models, we analyze the Urdu word distribution in different training datasets and compare the training methodologies. Finally, we uncover the specific translation issues and provide suggestions for improvements in Urdu machine translation systems.