Neeraj Sai Venkat Movva
Author directory2026
Does Quantization Erode Grounding? A Faithfulness Audit of On-Device LLM Compression
Neeraj Sai Venkat Movva | Sathvik Loke
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Neeraj Sai Venkat Movva | Sathvik Loke
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Weight quantization is the default route to running language models on consumer hardware, yet its effect on contextual faithfulness—a model’s willingness to ground answers in provided context rather than memorized priors—remains untested: faithfulness benchmarks use full-precision models, and quantization studies measure accuracy and perplexity, not grounding. We audit two instruction-tuned 3B models (Qwen2.5-3B, Llama-3.2-3B) across seven GGUF precisions from 16-bit to 2-bit, under aligned, counterfactual, and unanswerable contexts, using 300 SQuAD-derived items and three prompt templates per condition (900 trials per condition per precision). With an item-level clustered bootstrap we find faithfulness unchanged from 16 down to 5 bits in both models, while below that small but statistically detectable declines appear whose onset is model-dependent (4-bit for Llama, 3-bit for Qwen). Critically, production of the original, context-contradicting answer does not rise at any precision, so the intuitive "quantization erodes grounding" mechanism is not supported: lost faithfulness moves to off-target responses instead. Qwen additionally collapses into incoherent output at 2-bit while Llama stays fluent, making extreme compression a per-model question. Code and harness: https://github.com/movvaneeraj/quantfaith.