Low-rank (LR) pretraining underperforms full-rank (FR) pretraining, but why the gap persists remains unclear. Using GPT-2 on BabyLM, we show the representational split between them is not fixed at initialisation, spectrum-matched initialisation eliminates the representational gap at step zero, but it’s reasserted within a few training steps, implicating training dynamics over initialisation. We further show gradient dimensionality is a causally separable driver of this split, not merely a restatement of the weight factorization. Restricting or expanding gradient dimensionality of LR models directly shifts representational alignment with FR counterpart independent of weight rank. Finally, optimisation success and representational alignment dissociate, that is, spectral normalisation and other interventions improve perplexity without recovering FR-LR similarity. Also, we observered that representational alignment predicts downstream competence better than perplexity does. These results support gradient starvation as a genuine, partial mechanism, and argue efficient pretraining should be evaluated by both representational trajectory and performance, not solely the latter.

Scaling data and model size has driven recent advances in language modeling, but this strategy falters under scenarios with strict data constraints, as in the BabyLM Challenge. However, insights from Chinchilla highlights that smaller models trained on more data outperform larger counterparts trained inadequately, emphasizing the need for compact architectures. Furthermore, while embedding weight tying is a common parameter-saving technique, we find it significantly diminishes linguistic competence in compact models.In response, we explore alternative architectural strategies that preserve the parameter efficiency of tied models without sacrificing the representational benefits of untied embeddings. Consequently, we introduce SLlama a Llama3 architecture variant which incorporates targeted modifications—Repeated Reduced Hidden Size and Projection (RRHP), Permutated Weight Attention (PWA), Shared Projection Multi-Layer Perceptron (SPMLP), and Layer Weight Sharing—to compress Transformer components. Without relying on distillation, SLlama achieves a 31.72% improvement in linguistic knowledge acquisition over the BabyLlama baseline, with a comparable GLUE score and significantly lower parameter count. These results demonstrate that well-designed, compact models can rival larger ones under strict data constraints.