Reinforcement learning has driven the largest post-training gains on tasks with verifiable rewards, such as code and math. Extending this success to subjective tasks remains difficult: good responses must satisfy multiple soft criteria simultaneously, and no scalar reward can indicate which aspect of a response should improve. Personalization is a clear instance of this. A personalized response must balance helpfulness, user-specific relevance, and naturalness, yet standard verifiers cannot localize which specific dimension needs improvement. In this regime, supervised fine-tuning quickly saturates, and PPO with Bradley-Terry reward models is prone to reward hacking, especially via length bias, producing verbose or superficially personalized responses. We propose Critique-Post-Edit, an RL framework that combines (1) a personalized Generative Reward Model (GRM) producing multi-dimensional scores and natural-language critiques, and (2) a critique-guided editing step that turns feedback into nearby alternative responses for policy learning. The reward signal stays scalar; our contribution is to use personalized critiques to construct more informative local alternatives (responses close to the on-policy rollout in edit distance) during training. Under length-controlled evaluation, Critique-Post-Edit improves over PPO by 11.0 and 11.6 win-rate points on Qwen2.5-7B and 14B models, respectively. Notably, the 14B model matches or outperforms GPT-4.1 on the evaluated benchmarks. A static mechanism study further shows that critique-guided edits stay more local than extra rollouts and shift scores more favorably than generic rewrites. All of our experiments are on personalization; systematic evaluation on other subjective tasks is left to future work.