Zilu Huang
Author directory2026
Beyond SFT-Optimality: Cold-Start Data Selection for Reinforcement Learning via Gradient Subspace Coverage
Haisheng Lv | Zikun Su | Jingbang Liu | Zilu Huang | Xinzhang Liu | Wenmin Deng | Yu Zhao | Chao Wang | Shuangyong Song
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Haisheng Lv | Zikun Su | Jingbang Liu | Zilu Huang | Xinzhang Liu | Wenmin Deng | Yu Zhao | Chao Wang | Shuangyong Song
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Cold-start supervised fine-tuning (SFT) is a critical initialization step before reinforcement learning from verifiable rewards (RLVR). Yet existing cold-start data selection methods largely inherit criteria from supervised instruction tuning—sample quality, difficulty, perplexity, or diversity—implicitly assuming that a stronger SFT warm start is also a better starting point for RL. We argue that this assumption does not always hold in the SFT-to-RL pipeline: cold-start SFT not only reduces supervised loss but also shapes the initial update geometry that downstream reward optimization inherits. Building on this observation, we propose RL-Proxy-Coverage. Our method estimates a low-rank, reward-relevant gradient subspace from a small set of representative RL anchor prompts and verifier rewards, and selects, as the cold-start subset, those candidate SFT examples whose gradients provide non-redundant coverage of this subspace. In controlled mathematical-reasoning experiments, RL-Proxy-Coverage outperforms representative SFT-centric, hard-diverse, and reward-aware baselines at the early-to-mid RL checkpoints. Our findings suggest a new view of cold-start data: it should be regarded not merely as supervised examples for improving SFT, but as an initialization resource that shapes the starting point of downstream reward optimization.