Ningqi Chen
Author directory2026
HuggingBrew: Brewing High-Quality Mid-Training Data from the HuggingFace Community Datasets
Han Zhang | Ningqi Chen | Xiao Liu | Zhenghao Lin | Yeyun Gong | Yan Lu | Hanfang Yang
Findings of the Association for Computational Linguistics: EMNLP 2026
Han Zhang | Ningqi Chen | Xiao Liu | Zhenghao Lin | Yeyun Gong | Yan Lu | Hanfang Yang
Findings of the Association for Computational Linguistics: EMNLP 2026
Data curation for large language models (LLMs) has been extensively studied for both the pre-training and post-training stages, yet the intermediate mid-training phase—bridging general pretraining and task-specific fine-tuning—remains critically underexplored. Most existing mid-training data for state-of-the-art models remains proprietary. Meanwhile, the Hugging Face platform hosts a huge amount of community-contributed datasets, yet no systematic effort has been made to refine this rich ecosystem into a unified high-quality corpus. We present HuggingBrew, a large-scale mid-training corpus constructed through a principled, multi-step curation pipeline applied to the massive publicly accessible Hugging Face datasets. The pipeline operates at two complementary granularities: dataset-level operations act on each dataset independently, while corpus-level operations are applied jointly across all retained corpus. We construct three progressively curated variants—\textsf{Draught}, \textsf{Craft}, and \textsf{Reserve}—containing 1.6T, 450B, and 40B tokens, respectively. Continual pretraining experiments under a 100B-token budget show that series yields substantial improvements over baselines across mathematical reasoning, general knowledge, and code generation, validating the untapped value of community-contributed data for mid-training.