Reasoning LLMs-as-Judges, which benefit from inference-time scaling, offer a promising path for extending the success of reasoning models to non-verifiable domains, where output quality cannot be directly checked. While reasoning judges have shown gains on static evaluation benchmarks, their impact on actual LLM post-training remains underexplored. Therefore, we conduct a controlled study of non-reasoning and reasoning LLM-judges in reinforcement-learning-based LLM alignment. In our synthetic setting, a "gold" judge provides preference annotations for training smaller judges and evaluates the resulting policies. We find that non-reasoning judges lead to the standard reward hacking pattern: policies receive higher rewards from the training judge but lower rewards from the gold judge. Reasoning judges, in contrast, produce policies with high gold-judge rewards. However, a qualitative analysis shows that these high rewards arise from adversarial outputs that exploit the gold judge’s preferences and transfer to other LLM-judges on benchmarks like Arena-Hard. Together with our ablations on judge training design choices, these findings reveal both the promise and risks of using reasoning LLM-judges for non-verifiable LLM post-training.