Haowei Liu
Author directoryOther people with similar names: Haowei Liu
Unverified author pages with similar names: Haowei Liu
2026
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
Haowei Liu | Hsin-Tai Wu | Yi Fang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Haowei Liu | Hsin-Tai Wu | Yi Fang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Production text-to-SQL pipelines often end with an LLM-as-judge that nobody has checked against humans. We audited ours and found it broken: the deployed gpt-4o-mini judge agreed with two-author gold at Cohen’s κ = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, with most over-flags traceable to a single mechanism we call Grade-Hallucination. Replacing it with self-hosted Qwen3.6-27B (κ = 0.72) reaches the same range as Claude Opus 4.7 (κ = 0.71); the comparison is underpowered at n = 96, but the deployment decision is driven by Qwen costing about 1/300 per call. Ensembling is not a free win: pairing the weak judge with a stronger one degrades agreement; three strong judges with unanimity routing reach κ = 0.79 at 89.7% auto-coverage. Applying the audit recipe out-of-domain flags 25.5% of BIRD-financial’s expert-authored gold SQLs as likely substantive issues under our annotation protocol. Code and pre-registration are at https://anonymous.4open.science/r/synca-audit-CBE1; the datasets are deferred to the post-publication release for anonymization reasons.