High-quality preference data collection remains a bottleneck for RLHF reward model training: collecting multiple independent human annotations per sample is costly and hard to scale across domains and languages, while fully automated AI feedback (RLAIF) risks quality degradation. We present a production-grade hybrid human-AI annotation framework that introduces automated model-as-annotator pre-annotations into a quality-control (QC)-ratified workflow, contributing model votes only when automation is sufficiently reliable and leaving final adjudication to human QC. The pipeline (1) verifies factual claims with a web-search-grounded fact checker under a cleanroom constraint that keeps proprietary tokens out of training data, (2) elicits structured judgments from an ensemble of open-weights LLM judges, and (3) applies a two-call swap-consistency gate to mitigate position bias, with a data flywheel surfacing disagreements between QC and model outputs to drive improvement. In production, the framework reduces initial-labeler hours by ∼37%; across 11,092 production comparisons it reaches 81.6% agreement with QC (𝜅=0.63) and 82% to 88% on the public JudgeLM benchmark, with no QC degradation measured under an independent-model proxy. The resulting data supported a production reward model and aligned LLM.

Retrieval-augmented generation (RAG) offers a robust solution for developing enterprise internal virtual assistants by leveraging domain-specific knowledge and utilizing information from frequently updated corporate document repositories. In this work, we introduce the Enterprise-Knowledge RAG (EKRAG) dataset to benchmark RAG for enterprise knowledge question-answering (QA) across a diverse range of corporate documents, such as product releases, technical blogs, and financial reports. Using EKRAG, we systematically evaluate various retrieval models and strategies tailored for corporate content. We propose novel embedding-model (EM)-as-judge and ranking-model (RM)-as-judge approaches to assess answer quality in the context of enterprise information. Combining these with the existing LLM-as-judge method, we then comprehensively evaluate the correctness, relevance, and faithfulness of generated answers to corporate queries. Our extensive experiments shed light on optimizing RAG pipelines for enterprise knowledge QA, providing valuable guidance for practitioners. This work contributes to enhancing information retrieval and question-answering capabilities in corporate environments that demand high degrees of factuality and context-awareness.