Large-scale content moderation requires models that can apply complex, evolving policies under limited expert supervision. We present an empirical study of reinforcement learning (RL) for policy-grounded moderation across three production-derived tasks. Using established RL and inference primitives, we develop three integrity-specific designs: a policy-grounded full-trace reward, rationale-marginalized Monte Carlo scoring, and policy-structured reflection. Each design addresses an observed failure mode, including reasoning-length collapse and polarized confidence estimates. Across the three tasks, RL-Only systems trained on 200–836 labeled examples remain competitive with supervised fine-tuning (SFT) systems trained on thousands of examples, corresponding to up to approximately one order of magnitude greater label efficiency at the evaluated operating points. Performance generally improves with additional data and training compute before saturating, although larger rollout counts exhibit variability. The strongest SFT→RL systems attain point-estimate performance comparable to policy-trained full-time employee (FTE) reviewers across the three tasks.

Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta’s Llama-3-8B-Instruct model, which was trained on over 10M pairs.