Takahiro Shimizu
Author directory2026
Revisiting the Effectiveness of Word-level Tokenization in Language Model Pretraining
Takahiro Shimizu | Tianqi Wang | Jun Suzuki
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
Takahiro Shimizu | Tianqi Wang | Jun Suzuki
Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026)
Subword tokenization is widely used in modern large language models for open-vocabulary coverage and token efficiency. We revisit word-level tokenization by comparing it with SentencePiece Unigram under matched byte fallback settings, allowing both tokenizers to represent out-of-vocabulary units without unknown tokens. Using language-appropriate word-like segmentation, we evaluate models pretrained from scratch across six model configurations in English and, using a common model configuration, across four languages (English, French, Chinese, and Japanese), with vocabulary sizes of 10k, 50k, and 100k. Although subword tokenization is generally more token-efficient, raw accuracy varies across model configurations, languages, and vocabulary sizes, with word-level tokenization remaining competitive across the evaluated settings. Word-level tokenization also tends to achieve higher character-length-normalized accuracy, although this complementary metric does not isolate the effects of tokenization on model scoring or establish a universal advantage in language modeling quality. These results show that word-level tokenization can remain competitive with subword tokenization when open-vocabulary coverage and language-appropriate segmentation are provided. Our code and tokenizers are publicly available.