High-quality preference data collection remains a bottleneck for RLHF reward model training: collecting multiple independent human annotations per sample is costly and hard to scale across domains and languages, while fully automated AI feedback (RLAIF) risks quality degradation. We present a production-grade hybrid human-AI annotation framework that introduces automated model-as-annotator pre-annotations into a quality-control (QC)-ratified workflow, contributing model votes only when automation is sufficiently reliable and leaving final adjudication to human QC. The pipeline (1) verifies factual claims with a web-search-grounded fact checker under a cleanroom constraint that keeps proprietary tokens out of training data, (2) elicits structured judgments from an ensemble of open-weights LLM judges, and (3) applies a two-call swap-consistency gate to mitigate position bias, with a data flywheel surfacing disagreements between QC and model outputs to drive improvement. In production, the framework reduces initial-labeler hours by ∼37%; across 11,092 production comparisons it reaches 81.6% agreement with QC (𝜅=0.63) and 82% to 88% on the public JudgeLM benchmark, with no QC degradation measured under an independent-model proxy. The resulting data supported a production reward model and aligned LLM.

Inference-Time Scaling has been critical to the success of recent models such as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for inference-time scaling require tasks to have answers that can be verified, limiting their application to domains such as math, coding and logical reasoning. We take inspiration from how humans make first attempts, ask for detailed feedback from others and make improvements based on such feedback across a wide spectrum of open-ended endeavors. To this end, we collect HelpSteer3 data to train dedicated Feedback and Edit Models that are capable of performing inference-time scaling for open-ended general-domain tasks. In our setup, one model generates an initial response, which are given feedback by a second model, that are then used by a third model to edit the response. We show that performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo can be boosted by scaling the number of initial response drafts, effective feedback and edited responses. When scaled optimally, our setup based on 70B models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7 as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and DeepSeek R1 with 92.3.

Existing open-source helpfulness preference datasets do not specify what makes some responses more helpful and others less so. Models trained on these datasets can incidentally learn to model dataset artifacts (e.g. preferring longer but unhelpful responses only due to their length). To alleviate this problem, we collect HelpSteer, a multi-attribute helpfulness dataset annotated for the various aspects that make responses helpful. Specifically, our 37k-sample dataset has annotations for correctness, coherence, complexity, and verbosity in addition to overall helpfulness of responses. Training Llama 2 70B using the HelpSteer dataset with SteerLM technique produces a model that scores 7.54 on MT Bench, which is currently the highest score for open models that do not require training data from more powerful models (e.g. GPT-4). We release this dataset with CC-BY-4.0 license at https://huggingface.co/datasets/nvidia/HelpSteer