Ningqi Chen

Author directory

2026

Data curation for large language models (LLMs) has been extensively studied for both the pre-training and post-training stages, yet the intermediate mid-training phase—bridging general pretraining and task-specific fine-tuning—remains critically underexplored. Most existing mid-training data for state-of-the-art models remains proprietary. Meanwhile, the Hugging Face platform hosts a huge amount of community-contributed datasets, yet no systematic effort has been made to refine this rich ecosystem into a unified high-quality corpus. We present HuggingBrew, a large-scale mid-training corpus constructed through a principled, multi-step curation pipeline applied to the massive publicly accessible Hugging Face datasets. The pipeline operates at two complementary granularities: dataset-level operations act on each dataset independently, while corpus-level operations are applied jointly across all retained corpus. We construct three progressively curated variants—\textsf{Draught}, \textsf{Craft}, and \textsf{Reserve}—containing 1.6T, 450B, and 40B tokens, respectively. Continual pretraining experiments under a 100B-token budget show that series yields substantial improvements over baselines across mathematical reasoning, general knowledge, and code generation, validating the untapped value of community-contributed data for mid-training.