We introduce XpertGPT, a sparse decoder-only Transformer for data-constrained language modelling that routes tokens to parallel expert pathways operating at multiple contextual scales. Under the BabyLM strict-small budget of roughly ten million words, conventional dense Transformers spend identical computation on every token and impose a single fixed receptive field. Each XpertGPT layer pairs a dense global-attention stream with four sparse sliding-window expert branches, whose attention spans range from 64 down to 4 tokens, chosen by Expert-Choice routing that balances expert load by construction without any auxiliary loss, and applies rotary position embeddings and gated feed-forward networks throughout. Trained on the BabyLM 2026 Strict-Small corpus, our 52.6M-parameter model reaches an overall average of 38.42, surpassing the official GPT-2(98.4M) baseline (37.38) across grammatical acceptability, world-knowledge, and language-understanding benchmarks.

Recent advancements in self-improvement for Large Language Models (LLMs) have efficiently enhanced model capabilities without significantly increasing costs, particularly in terms of human effort. While this area is still relatively young, its extension to the multimodal domain holds immense potential for leveraging diverse data sources and developing more general self-improving models. This survey is the first to provide a comprehensive overview of self-improvement in Multimodal LLMs (MLLMs). We provide a structured overview of the current literature and discuss methods from three perspectives: 1) data collection, 2) data organization, and 3) model optimization, to facilitate the further development of self-improvement in MLLMs. We also include commonly used evaluations and downstream applications. Finally, we conclude by outlining open challenges and future research directions.