Hunter Tuerlings
Author directory2026
Verbalized Uncertainty in Coding LLMs: A Contamination-Aware Study of Confidence Calibration
Hunter Tuerlings | Matias Valdenegro-Toro
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Hunter Tuerlings | Matias Valdenegro-Toro
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Coding large language models (LLMs) can produce plausible solutions that fail hidden test cases, making it difficult to judge when to trust their output. This paper presents a contamination-aware comparison of established methods for eliciting verbalized confidence in generated code. Six instruction-tuned, coding-specialized LLMs from three model families are evaluated on 365 problems from a shared contamination-aware LiveCodeBench window. For each frozen code generation, confidence is elicited separately under three strategies, Direct numeric scoring, Chain-of-Thought reasoning followed by a score, and Linguistic confidence labels. Under strict scoring, all 18 model-strategy combinations have negative Net Calibration Error, showing a consistent tendency toward overconfidence. Linguistic elicitation achieves the lowest Expected Calibration Error for four of six models, but several favourable values result from collapsed confidence distributions. Eight of the 18 combinations are collapsed and five are unstable across repeated runs, including three Chain-of-Thought combinations. When outputs are ranked by confidence and the least confident are reviewed first, Direct elicitation gives the largest improvement over random selection in four of six models. Verbalized confidence cannot be reduced to a single trustworthy pass probability; calibration, ranking ability, distributional collapse, and stability must each be checked separately.