We present our submission to the Strict-Small track of the 2026 BabyLM Challenge. We study whether a developmentally-inspired curriculum, particularly an adaptive teacher-feedback curriculum based on the student’s own productions, can improve language model pre-training under a 10M word budget. We trained a 58M-parameter student from scratch under five separate conditions that shared the same architecture and tokenizer, isolating different aspects of the training data and curriculum structure. We built an official baseline trained on a cleaned version of the official Strict-Small corpus and also a synthetic baseline trained on teacher model generated data. Then, we developed a fixed curriculum to train a model on the same synthetic child-directed text in three developmental stages: infant, toddler, and child. Finally, we introduced an adaptive feedback curriculum in which the teachers generate additional training data by correcting the student’s productions during training, which adapts the training data according to the learner’s level of understanding. The fixed curriculum demonstrated superior performance on the BLiMP-Supplement, EWoK, and COMPS benchmarks, while the adaptive feedback curriculum substantially improved the Entity Tracking performance in one configuration. However, our official baseline model achieved the highest BLiMP, SuperGLUE, and overall macro-average scores, indicating that our curriculum methods did not produce the overall improvements that we had hypothesized. Our code is available at https://github.com/dcgaines/babylm-2026.

Users of Augmentative and Alternative Communication (AAC) may write letter-by-letter via an interface that uses a character language model. However, most state-of-the-art large pretrained language models predict subword tokens of variable length. We investigate how to practically use such models to make accurate and efficient character predictions. Our algorithm for producing character predictions from a subword large language model (LLM) provides more accurate predictions than using a classification layer, a byte-level LLM, or an n-gram model. Additionally, we investigate a domain adaptation procedure based on a large dataset of sentences we curated based on scoring how useful each sentence might be for spoken or written AAC communication. We find our procedure further improves model performance on simple, conversational text.

In this paper we detail our submissions to the Strict and Strict-Small tracks of the 2024 BabyLM Challenge. We approach this challenge with two methodologies: i) use of a novel dataset, and ii) development of a pre-training technique based on the fusion of child language acquisition with traditional masked language modeling, which we call curriculum masking. The novel dataset used for this task is based on user submissions to the Reddit forum (i.e., subreddit) “Explain Like I’m Five”, which explains diverse concepts using simple language. Curriculum masking works by creating learning phases based on a standard child language development timeline, where the masked words learned by the model start with simple nouns and gradually expand to include more complex parts of speech. We show that using internet-based training data shows a small improvement in evaluation scores as compared to baseline training data. Our proposed pre-training method of curriculum masking is conceptually novel and also shows improved rates of learning over typical masked language modeling pre-training, potentially allowing for good performance with fewer total epochs on smaller training datasets. Code for the curriculum masking implementation is shared at https://github.com/evan-person/curriculumMaskingBabyLM2024.