Mohammad Saquib
Author directory2026
Performance of Grapheme-Based Tokenizers on Word-level and Natural Language Understanding tasks for Indic Languages
Shagun Dwivedi | Mohammad Saquib | Kaushik Gopalan
Findings of the Association for Computational Linguistics: EMNLP 2026
Shagun Dwivedi | Mohammad Saquib | Kaushik Gopalan
Findings of the Association for Computational Linguistics: EMNLP 2026
Tokenization is one of the factors which influences the multilingual downstream performance of a language model. This paper demonstrates that vocabulary constrained grapheme cluster based tokenization leads to improved performance on Natural Language Understanding and Word-level tasks. We assess three different grapheme cluster based tokenizers, which are the Grapheme Cluster tokenizer, the Grapheme Pair Encoding, and a vocabulary constrained version of Grapheme Pair Encoding, across eleven Indic languages, namely Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu. We present the intrinsic performance of three grapheme cluster based tokenizers using Fertility, Rényi Efficiency, and Percentile Frequency. We also compare the performance of these grapheme based tokenizers with seven popular multilingual and Indic-focused tokenizers on downstream tasks. We find that the vocabulary constrained Grapheme Pair Encoding tokenizer exhibits competitive and sometimes superior downstream performance in comparison to other grapheme cluster based tokenizers.