Reasoning LLMs-as-Judges, which benefit from inference-time scaling, offer a promising path for extending the success of reasoning models to non-verifiable domains, where output quality cannot be directly checked. While reasoning judges have shown gains on static evaluation benchmarks, their impact on actual LLM post-training remains underexplored. Therefore, we conduct a controlled study of non-reasoning and reasoning LLM-judges in reinforcement-learning-based LLM alignment. In our synthetic setting, a "gold" judge provides preference annotations for training smaller judges and evaluates the resulting policies. We find that non-reasoning judges lead to the standard reward hacking pattern: policies receive higher rewards from the training judge but lower rewards from the gold judge. Reasoning judges, in contrast, produce policies with high gold-judge rewards. However, a qualitative analysis shows that these high rewards arise from adversarial outputs that exploit the gold judge’s preferences and transfer to other LLM-judges on benchmarks like Arena-Hard. Together with our ablations on judge training design choices, these findings reveal both the promise and risks of using reasoning LLM-judges for non-verifiable LLM post-training.

Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of the generated critiques.