G.L. Salvin
Author directory2026
Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation
G.L. Salvin | Swapnil Hingmire
Proceedings of the Eleventh Conference on Machine Translation
G.L. Salvin | Swapnil Hingmire
Proceedings of the Eleventh Conference on Machine Translation
COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts. Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target. We test it on IndicMT Eval by re-encoding the target into Latin script, which changes orthographic form while holding content and human ratings fixed. Script identity then accounts for 22.9% of native-script COMET variance, and agreement with annotators falls in all five languages studied. We trace the effect to the tokeniser and measure it with three label-free diagnostics. The bias is two faults, not one. Scores from different scripts occupy incompatible ranges, and within a single script the metric orders translations less accurately. No order-preserving transform of the score can repair the second fault. The first is removed exactly by COMET-QN, which maps the score distribution of each (language, script) pair onto a shared reference. Pooled agreement with annotators rises from 0.300 to 0.399, which is what makes scores from different scripts safe to place on one axis, and every within-language ordering is provably preserved. A regressor over parity features recovers a further 17.1% of the lost sensitivity. The remainder belongs to the encoder, and no post-processing can reach it. We therefore recommend publishing the normalised score, the three diagnostics, and the identity of the tokeniser they were computed against, so that a reader can tell how much of a score reflects translation quality and how much reflects the writing system.
DoDS-IITPKD: LoRA Fine-Tuning and LLM Post-Editing for Low-Resource Indic Machine Translation
G.L. Salvin | Amisha Ramesan | Abigairl Chigwededza | Shrikant Budde | Swapnil Hingmire
Proceedings of the Eleventh Conference on Machine Translation
G.L. Salvin | Amisha Ramesan | Abigairl Chigwededza | Shrikant Budde | Swapnil Hingmire
Proceedings of the Eleventh Conference on Machine Translation
We describe the DoDS-IITPKD submissions to the WMT 2026 Shared Task on Low-Resource Indic Language Translation. We cover six English-centric pairs in both directions: Assamese, Mizo, Khasi, and Manipuri in Category 1, and Bodo and Kokborok in Category 2. Each system adapts a frozen multilingual backbone with low-rank adapters (LoRA with DoRA and rank-stabilised scaling). NLLB-200-3.3B handles Mizo, Khasi, and Kokborok, and IndicTrans2-1B handles Assamese, Manipuri, and Bodo. For the two languages outside the NLLB tokenizer (Khasi and Kokborok) we use same-script surrogate language tags. We add self back-translation for the Indic-English directions and, for some systems, a line-aligned LLM post-editing pass. On the official test sets, our systems obtain the best BLEU of any submission for Bodo to English (36.10) and English to Kokborok (7.36), and the best on time BLEU for Kokborok to English (20.28). LLM post-editing gives large gains in the Indic-English direction (up to +8.20 BLEU) and almost none in the English-Indic direction.