Xinyu Shao
Author directory2026
GRAIL: Gradient-Reward Aligned Initialization for LoRA-based GRPO
Liwen Shi | Zijie Zeng | Bowen Wang | Yunji | Zeyu Chen | Xinyu Shao | Xinzheng Sun | Xuyang Xiong | Haoyang Wu | Sheng Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Liwen Shi | Zijie Zeng | Bowen Wang | Yunji | Zeyu Chen | Xinyu Shao | Xinzheng Sun | Xuyang Xiong | Haoyang Wu | Sheng Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Group Relative Policy Optimization (GRPO) improves LLM reasoning, but LoRA-based GRPO can underperform when the initialized adapter subspace is poorly aligned with reward-induced gradients. We propose GRAIL, Gradient-Reward Aligned Initialization for LoRA-based GRPO, which estimates reward-aligned gradients from grouped rollouts before policy optimization and uses them to construct LoRA factors. The full GRAIL configuration includes a reference-solution fallback only for all-zero-correctness groups during calibration; after initialization, all models use the standard GRPO objective without extra trainable parameters or loss changes. We also study Degenerate-Group Reflection Resampling (DGRR) as an optional extension that resamples all-zero-correctness groups after self-reflection without exposing reference solutions. Across Qwen2.5, Phi-3.5, and Gemma-2 backbones on GSM8K and MATH500, GRAIL consistently improves LoRA-based GRPO, with particularly strong gains on smaller Qwen and Gemma models. Analyses attribute these gains to stronger GRPO-gradient capture, smaller principal angles, and better handling of correctness-degenerate groups.