Archie Chaudhury
Author directory2026
But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors
Leon Eshuijs | Archie Chaudhury | Alan McBeth | Ethan Nguyen
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Leon Eshuijs | Archie Chaudhury | Alan McBeth | Ethan Nguyen
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
LLM-as-a-judge is widely used as a scalable substitute for human evaluation, yet judges are least reliable on subtle dishonesty such as sycophancy and manipulation, where a judge scoring a response in isolation lacks a reference point. We introduce Judge Using Safety-Steered Alternatives (JUSSA), a framework that reduces this evaluator uncertainty using the assessed model’s own internal representations. An honesty-promoting steering vector, optimized from a single training example, generates a contrastive alternative against which each response is judged. On a new manipulation dataset with human-validated response pairs, JUSSA improves detection for both GPT-4.1 (AUROC 0.893→0.946) and Claude Haiku (0.859→0.929) judges. The gains concentrate almost entirely on the pairs a judge would otherwise misrank and reverse once a judge has saturated a task, so contrastive evaluation helps when the judge retains headroom rather than as a function of model size. A layer-wise analysis further locates steering effectiveness in the middle layers, where model representations of honest and dishonest prompt processing diverge. Our work demonstrates that steering vectors can serve as tools for evaluation rather than for improving model outputs at inference, opening a new direction for white-box auditing.