Saurabh Chauhan
Author directory2026
Script Is Not Preprocessing: Tokenizer-Mediated Transfer in Endangered Language Machine Translation
Pooja Singh | Saurabh Chauhan | Nishant Agnihotri | Sandeep Kumar
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Pooja Singh | Saurabh Chauhan | Nishant Agnihotri | Sandeep Kumar
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Multilingual machine translation for endangered and severely underrepresented languages faces two interacting challenges: scarce parallel data and uneven subword vocabulary allocation across writing systems. We study these factors for seven Indian tribal languages, Gondi, Mundari, Bhili, Santali, Kui, Garo, and Kokborok, spanning four language families and four scripts. We release a speaker-curated Hindi–English–X tri-parallel corpus of approximately 20k sentences per language across six community-relevant domains, with all development and test references human-curated or independently speaker-verified. We evaluate prompted LLMs, fine-tuned IndicTrans2 and NLLB-600M, and byte-level ByT5 under corpus-Native, Devanagari-Unified, and Latin-Unified script conditions. Fine-tuning improves CHRF++ by 16-25 points over prompting, but script unification is strongly model-dependent: the same Latin conversion improves Kui under NLLB-600M while substantially degrading performance under IndicTrans2. We introduce Script Tokenization Parity and Script Sensitivity to analyse script–tokenizer interaction; byte-level modelling reduces Script Sensitivity from 14.2 to 2.1 CHRF++, localizing most cross-script variance in learned vocabulary allocation. Our findings show that script choice in endangered-language MT is not routine preprocessing, but a community-sensitive modelling decision that directly shapes linguistic representation, translation quality, and equitable digital access.