Ananto Nayan Bala
Author directory2026
Does Quantization Change What Agreement Means? A Paired Audit of Homogeneous LLM Panels
Ananto Nayan Bala
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Ananto Nayan Bala
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Agreement among repeated large language model (LLM) responses is often used as a confidence signal. We test whether that signal changes after quantization by comparing five-response panels from the same checkpoints at FP16 and 4-bit NF4. We evaluate two open models on 300 TruthfulQA multiple-choice questions and 300 GSM8K problems, matching prompts, sampling settings, and seeds. Under strict parsing, quantization reduces GSM8K panel accuracy by 10.5 percentage points for Qwen2.5-3B and 6.3 for Phi-3.5-mini. The probability that all five responses are wrong rises by 9.0 and 5.0 points. Agreement also becomes less useful for selecting reliable answers, but all-wrong panels do not consistently become more likely to give one identical answer. Relaxed parsing preserves the selective-reliability result for both models, although Phi’s panel-accuracy and all-wrong differences become inconclusive. On TruthfulQA, Phi degrades while Qwen is largely stable. Quantization can therefore reduce panel reliability without consistently pushing all members toward the same wrong answer.