Large Language Models (LLMs) have demonstrated strong reasoning capabilities, yet achieving nuanced personality alignment for simulating human conversation remains challenging. This difficulty stems from two bottlenecks: the scarcity of scalable, privacy-respecting training data and the limited ability of conventional reinforcement learning (RL) rewards to capture subtle personality-relevant signals. In this work, we introduce a unified framework to address these issues. First, we propose the Personality Alignment Choice Forge (PACF), an automatic pipeline that transforms existing conversational datasets into structured, multiple-choice benchmarks, constraining the model’s action space and enabling scalable evaluation grounded in observed human choices. Second, we present Group Sentence-Specified Sequence Policy Optimization (GSSPO), an RL algorithm designed to address the coarse credit assignment problem in methods that apply a uniform reward across all tokens. By integrating stable sequence-level updates with differentiated, rubric-guided token-level feedback, GSSPO assigns reward signals to more localized spans of the response. This targeted approach mitigates optimization instabilities caused by undifferentiated reward signals, leading to more effective training in our experiments. On the Qwen3 model family, GSSPO achieves the best personality-alignment accuracy on every benchmark and scale in the SFT+RL regime, and leads the other RL methods on five of six in the SFT-Free regime. These results suggest a scalable route toward training LLMs that better match measured personality-alignment preferences. Our core code is available at https://github.com/HKUST-FintechLab/GSSPO