Naive joint training of large language models (LLMs) for multilingual preference alignment can suffer from negative interference. This is a known issue in multilingual training, where conflicting objectives degrade overall performance. However, the impact of this phenomenon in the context of multilingual preference alignment remains largely underexplored. To address this issue, we propose ConGrad, an effective and scalable filtering method that mitigates this interference by identifying and selecting preference samples that exhibit high cross-lingual affinity. Based on principles of multi-objective optimization, our approach computes an aggregated, cross-lingually beneficial gradient direction and uses this to filter for samples whose individual gradients align with this consensus direction. To ensure scalability for LLMs, we incorporate a sublinear gradient compression strategy that reduces memory overhead during gradient accumulation. We integrate ConGrad into a self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages. Results show that ConGrad consistently outperforms strong baselines in both seen and unseen languages, with minimal alignment tax.

Supervised fine-tuning (SFT) is typically viewed as behavior cloning on expert demonstrations. Recent methods recover richer supervision from demonstrations through self-play preference synthesis or bilevel inverse reinforcement learning, yet these require additional optimization stages or separately trained reward models. Motivated by the connection between SFT and inverse soft-Q learning, we propose Dense-Path REINFORCE (DPR), a lightweight post-SFT policy-improvement method that extracts a dense reward signal directly from the SFT training trajectory. We construct a token-level reward from the log-probability gap between the final SFT policy and an intermediate checkpoint, capturing the incremental likelihood gain acquired during training while mitigating the length and early-EOS biases inherent in raw log-probability rewards. DPR reuses the same demonstrations and prompts as SFT, requires no preference data, no external teacher, and no separately trained reward model. Across four pretrained backbones and four instruction-following evaluations, DPR obtains 55.2-66.8% win rates over SFT, improves MT-Bench by 0.22-0.46 points, and matches or exceeds the best demonstration-only baseline in 15 of 16 settings.