Saket Reddy
Author directory2026
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
Saket Reddy | Ke Yang | ChengXiang Zhai
Findings of the Association for Computational Linguistics: ACL 2026
Saket Reddy | Ke Yang | ChengXiang Zhai
Findings of the Association for Computational Linguistics: ACL 2026
Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, social bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates. In this paper, we propose BiasGRPO, an adaptation of Group Relative Policy Optimization (GRPO) that stabilizes alignment by normalizing rewards across a group of sampled completions. By substituting the value function with a group-relative baseline, our approach reduces instability while maintaining the exploration benefits of online reinforcement learning. To adapt GRPO, we curate and synthetically extend a dataset spanning multiple domains and contexts, and create a custom, bias-specific reward model for effectively guiding generation while avoiding knowledge degradation. We find that BiasGRPO outperforms DPO and PPO across multiple benchmarks, indicating its effectiveness as an alignment technique that can overcome the limitations of previous preference-based methods.
A Geometric Perspective on Stabilizing Value Conflict Resolution
Saket Reddy | Andy Liu
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Saket Reddy | Andy Liu
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Large Language Models (LLMs) often struggle to navigate user requests that contain complex value conflicts, as direct scalar rewards are often ill-equipped to handle these scenarios. To address this challenge, we investigate how chain-of-thought (CoT) reasoning can help improve performance in this domain. Geometrically, we show that CoT correlates with further smoothing the model’s loss landscape in its sharpest direction, helping resolve the optimization instability of traditional scalar rewards. In terms of performance, we demonstrate via relevant downstream benchmarks that CoT may increase capability, demonstrating that value conflict-focused CoT has the potential to be an effective mechanism for better moral reasoning. To capitalize on this potential, we create a new value conflict-focused CoT design that correlates with further smoothing the sharpest direction of the loss landscape and further increases moral reasoning performance. This finding shows that explicitly modifying and improving the design of reasoning dynamics offers a promising avenue for advancing pluralistic alignment in LLMs.