Long Huang
Author directory2026
GRADFILTERING: Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning
Zhihang Yuan | Chengyu Yue | Long Huang | Litu Ou | Lei Shi
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zhihang Yuan | Chengyu Yue | Long Huang | Litu Ou | Lei Shi
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Modern instruction-tuning mixtures are increasingly large, redundant, and heterogeneous, making full-data supervised fine-tuning costly and often unnecessary. We propose GRADFILTERING, an objective-agnostic, validation-free data selection method that scores examples using uncertainty-aware proxy training dynamics. GRADFILTERING trains a lightweight proxy model with a LoRA ensemble and ranks examples by a Gradient Signal-to-Noise Ratio (G-SNR), which combines early-to-late gradient drop with late-stage ensemble disagreement. Because selection requires neither target validation examples nor target-model gradients, the selected subsets can be reused across downstream targets. Across Alpaca and Tulu-3-52K, dense LLaMA-2-7B, LLaMA-3-8B, and Qwen3-8B targets, and a Qwen3-30B-A3B MoE target, GRADFILTERING-selected 5–15% subsets consistently outperform random selection and are competitive with the target-aware LESS-T reference while remaining validation-free. We further evaluate broad model capabilities on objective benchmarks and analyze proxy choice, gradient windows, ensemble size, human preferences, and selection cost, demonstrating the practical utility of early proxy dynamics for reusable, compact instruction-tuning data selection. Code is available at this GitHub repository.