We present three BabySteps systems that strengthen baselines for data-constrained pretraining in the strict-small (10M words) and strict (100M words) NLP tracks of the 2026 BabyLM Challenge. Across our GPT-BERT experiments, optimizer choice and learning-rate tuning produced our largest gains, highlighting the importance of well-tuned baselines for evaluating progress in sample-efficient pretraining. The curriculum, synthetic-data, teacher-supervision, and corpus-mixing interventions that we tested did not improve the final aggregate score.

We also test whether the official GPT-2 checkpoint is a sufficiently strong reference for causal language modeling. A 127.6M-parameter Llama-style decoder trained with Muon and Liger-optimized kernels obtains a seven-suite NLP average of 56.18, compared with 52.79 for our completed reevaluation of the official BabyLM GPT-2 strict checkpoint, and scores higher on all seven categories. This is not a controlled architectural ablation: architecture, optimizer, implementation, and hyperparameters change together. Instead, it provides a stronger practical causal baseline. In addition, we report failed data interventions, a numerical failure caused by batches with no supervised target tokens, and substantial variance between small evaluation suites.