<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="basit-azeemi-2026-psmt">
    <titleInfo>
        <title>PSMT-eval: A Benchmark Dataset for Machine Translation Evaluation in Punjabi Shahmukhi</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Abdul</namePart>
        <namePart type="family">Basit</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Abdul</namePart>
        <namePart type="given">Hameed</namePart>
        <namePart type="family">Azeemi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Thirteenth Workshop on Asian Translation (WAT 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Toshiaki</namePart>
            <namePart type="family">Nakazawa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isao</namePart>
            <namePart type="family">Goto</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hengqin, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-446-0</identifier>
    </relatedItem>
    <abstract>This paper introduces PSMT-eval, a new dataset for machine translation (MT) evaluation in Punjabi Shahmukhi, a low-resource language spoken by over 80 million people. We constructed an English–Punjabi Shahmukhi parallel corpus through pivot-based human translation, where the Urdu side of existing English–Urdu datasets was manually translated into Punjabi Shahmukhi. Four widely used English–Urdu datasets were selected: tatoeba-test.eng-urd, FLORES101, IN22, and MKB, with total count of 5182 sentences. To assess the dataset, we evaluated translations generated by state-of-the-art large language models (LLMs), including GPT-5, GPT-OSS-120B, Claude 4 Sonnet, LLaMA-3.3 70B Instruct, and Qwen-3 32B. Results show that only Claude 4 Sonnet achieved competitive performance, with a maximum SacreBLEU score of 17.51 on IN22. GPT-5 followed with a maximum SacreBLEU score of 10, while other models performed poorly. Further analysis highlights major challenges in Punjabi Shahmukhi MT. To the best of our knowledge, this is the first benchmark dataset specifically for MT evaluation in Punjabi Shahmukhi. PSMT-eval is a valuable resource for advancing low-resource MT research and is publicly available to support future work.</abstract>
    <identifier type="citekey">basit-azeemi-2026-psmt</identifier>
    <location>
        <url>https://aclanthology.org/2026.wat-1.1/</url>
    </location>
    <part>
        <date>2026-11</date>
        <extent unit="page">
            <start>1</start>
            <end>8</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T PSMT-eval: A Benchmark Dataset for Machine Translation Evaluation in Punjabi Shahmukhi
%A Basit, Abdul
%A Azeemi, Abdul Hameed
%Y Nakazawa, Toshiaki
%Y Goto, Isao
%S Proceedings of the Thirteenth Workshop on Asian Translation (WAT 2026)
%D 2026
%8 November
%I Association for Computational Linguistics
%C Hengqin, China
%@ 979-8-89176-446-0
%F basit-azeemi-2026-psmt
%X This paper introduces PSMT-eval, a new dataset for machine translation (MT) evaluation in Punjabi Shahmukhi, a low-resource language spoken by over 80 million people. We constructed an English–Punjabi Shahmukhi parallel corpus through pivot-based human translation, where the Urdu side of existing English–Urdu datasets was manually translated into Punjabi Shahmukhi. Four widely used English–Urdu datasets were selected: tatoeba-test.eng-urd, FLORES101, IN22, and MKB, with total count of 5182 sentences. To assess the dataset, we evaluated translations generated by state-of-the-art large language models (LLMs), including GPT-5, GPT-OSS-120B, Claude 4 Sonnet, LLaMA-3.3 70B Instruct, and Qwen-3 32B. Results show that only Claude 4 Sonnet achieved competitive performance, with a maximum SacreBLEU score of 17.51 on IN22. GPT-5 followed with a maximum SacreBLEU score of 10, while other models performed poorly. Further analysis highlights major challenges in Punjabi Shahmukhi MT. To the best of our knowledge, this is the first benchmark dataset specifically for MT evaluation in Punjabi Shahmukhi. PSMT-eval is a valuable resource for advancing low-resource MT research and is publicly available to support future work.
%U https://aclanthology.org/2026.wat-1.1/
%P 1-8

Markdown (Informal)

[PSMT-eval: A Benchmark Dataset for Machine Translation Evaluation in Punjabi Shahmukhi](https://aclanthology.org/2026.wat-1.1/) (Basit & Azeemi, WAT 2026)

• PSMT-eval: A Benchmark Dataset for Machine Translation Evaluation in Punjabi Shahmukhi (Basit & Azeemi, WAT 2026)

ACL

• Abdul Basit and Abdul Hameed Azeemi. 2026. PSMT-eval: A Benchmark Dataset for Machine Translation Evaluation in Punjabi Shahmukhi. In Proceedings of the Thirteenth Workshop on Asian Translation (WAT 2026), pages 1–8, Hengqin, China. Association for Computational Linguistics.