Toshiki Yamaguchi

Author directory

2026

Subword tokenization is widely used to address the out-of-vocabulary problem in natural language processing. However, frequency-based subword tokenization can produce artifacts in the resulting vocabulary. Beyond the well-known junk tokens (meaningless fragments), we introduce the notion of subsumed tokens, valid words whose corpus frequencies are suppressed by longer tokens that contain them. Together with fragment tokens, these are referred to as overlapping tokens. We propose a post-processing method that reduces fragment tokens and restores subsumed tokens. The method searches over possible binary segmentations of character sequences and selects tokenizations that maximize corpus frequency, while optionally assigning bonuses to word spans identified by a morphological analyzer. As a post-processing step, the method can be applied to existing tokenizers such as Byte Pair Encoding (BPE) and the Unigram Language Model (ULM) without rebuilding their vocabularies. Although evaluated on Japanese, the approach may also be applicable to other languages given an appropriate morphological analyzer. Experiments on the Wiki-40B Japanese corpus show that our method reduces overlapping tokens, leading to more reliable statistics for valid words, and improves representation quality while preserving downstream performance across tasks.