Jesse-Paul Osemeke

Author directory

2026

Continued pretraining has become a widely adopted approach for improving multilingual large language models, consistently improving performance across a diverse range of downstream NLP tasks. While these gains are well documented, little is known about how continued pretraining reshapes the internal representations that give rise to these improvements. In this work, we investigate how African-language continued pretraining reorganizes the internal representation space of multilingual language models. We compare three pretrained multilingual models, XLM-R, Gemma-3, and Llama-3.1, with their African-adapted counterparts, AfroXLMR, AfriqueGemma, and AfriqueLlama, spanning both encoder-only and decoder-only architectures. Using layer-wise representation similarity, representation geometry, linear probing, and cross-lingual transfer analyses, we characterize how continued pretraining reorganizes multilingual representations across encoder-only and decoder-only architectures. Our analysis shows that continued African-language pretraining selectively reorganizes multilingual representations rather than uniformly modifying them. This reorganization is architecture dependent, expands the effective dimensionality of the representation space, and improves both the accessibility of linguistic knowledge and its cross-lingual transfer across African languages. Together, these findings provide new insights into how continued pretraining reshapes multilingual representation spaces and advances our understanding of multilingual adaptation for underrepresented languages.