We introduce XpertGPT, a sparse decoder-only Transformer for data-constrained language modelling that routes tokens to parallel expert pathways operating at multiple contextual scales. Under the BabyLM strict-small budget of roughly ten million words, conventional dense Transformers spend identical computation on every token and impose a single fixed receptive field. Each XpertGPT layer pairs a dense global-attention stream with four sparse sliding-window expert branches, whose attention spans range from 64 down to 4 tokens, chosen by Expert-Choice routing that balances expert load by construction without any auxiliary loss, and applies rotary position embeddings and gated feed-forward networks throughout. Trained on the BabyLM 2026 Strict-Small corpus, our 52.6M-parameter model reaches an overall average of 38.42, surpassing the official GPT-2(98.4M) baseline (37.38) across grammatical acceptability, world-knowledge, and language-understanding benchmarks.