Script Is Not Preprocessing: Tokenizer-Mediated Transfer in Endangered Language Machine Translation

Pooja Singh, Saurabh Chauhan, Nishant Agnihotri, Sandeep Kumar


Abstract
Multilingual machine translation for endangered and severely underrepresented languages faces two interacting challenges: scarce parallel data and uneven subword vocabulary allocation across writing systems. We study these factors for seven Indian tribal languages, Gondi, Mundari, Bhili, Santali, Kui, Garo, and Kokborok, spanning four language families and four scripts. We release a speaker-curated Hindi–English–X tri-parallel corpus of approximately 20k sentences per language across six community-relevant domains, with all development and test references human-curated or independently speaker-verified. We evaluate prompted LLMs, fine-tuned IndicTrans2 and NLLB-600M, and byte-level ByT5 under corpus-Native, Devanagari-Unified, and Latin-Unified script conditions. Fine-tuning improves CHRF++ by 16-25 points over prompting, but script unification is strongly model-dependent: the same Latin conversion improves Kui under NLLB-600M while substantially degrading performance under IndicTrans2. We introduce Script Tokenization Parity and Script Sensitivity to analyse script–tokenizer interaction; byte-level modelling reduces Script Sensitivity from 14.2 to 2.1 CHRF++, localizing most cross-script variance in learned vocabulary allocation. Our findings show that script choice in endangered-language MT is not routine preprocessing, but a community-sensitive modelling decision that directly shapes linguistic representation, translation quality, and equitable digital access.
Anthology ID:
2026.emnlp-main.1715
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
39110–39150
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1715/
DOI:
Bibkey:
Cite (ACL):
Pooja Singh, Saurabh Chauhan, Nishant Agnihotri, and Sandeep Kumar. 2026. Script Is Not Preprocessing: Tokenizer-Mediated Transfer in Endangered Language Machine Translation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 39110–39150, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Script Is Not Preprocessing: Tokenizer-Mediated Transfer in Endangered Language Machine Translation (Singh et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1715.pdf