Olga Shcherbakova

Author directory

2026

Tokenization is a central component of pretrained language models, especially for morphologically rich low-resource languages. We present a controlled study of morphology-aware tokenization for Belarusian masked language models. To isolate the effect of tokenization, we pretrain three RoBERTa-family models on the same Belarusian HPLT corpus, varying only the tokenizer: standard WordPiece, WordPiece trained on morpheme-segmented text, and a dictionary-based hybrid morpheme tokenizer with WordPiece fallback. We evaluate the models on masked language modeling, Universal Dependencies morphology-sensitive tasks, and BelarusianGLUE. Morphology-aware tokenization substantially improves morpheme-boundary preservation and yields clear gains on explicitly morphological tasks, especially fine-grained feature prediction and lemmatization, where models must recover information encoded in inflectional structure. However, these gains do not uniformly transfer to broad sentence-level NLU: on BelarusianGLUE, morphology-aware tokenizers improve some tasks but remain comparable to or weaker than standard WordPiece on others. These findings suggest that morpheme-informed tokenization is a targeted inductive bias rather than a universal replacement for statistical subword segmentation.