Srutanik Bhaduri
Author directory2026
Latent Truth Decoding: Training Free Mitigation of Sycophancy in Small Language Models
Srutanik Bhaduri | Neelesh Kumar Shukla | Viji Krishnamurthy
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Srutanik Bhaduri | Neelesh Kumar Shukla | Viji Krishnamurthy
Proceedings of the 1st Workshop on Grounding Language Models: Learning Faithfully and Efficiently (GroundLM 2026)
Small Language Models (SLMs) under 10B parameters are increasingly deployed on edge devices for privacy-sensitive, latency-critical applications, where their factual reliability is pivotal and no larger model is available to fall back on. These SLMs frequently exhibit sycophancy: they abandon a correct answer the moment a user confidently asserts an incorrect one. In this paper, we study where this failure originates and how to fix it without retraining. We introduce a prefix/suffix multiple-choice construction that reduces every question to a single answer token, enabling layer-wise analysis of exactly when a model commits to the truth or the lie. Using this probe, we show that sycophancy is pervasive and, crucially, that it is a routing failure rather than a knowledge failure, i.e. middle layers retrieve the correct answer with a strong, linearly separable signal but the final alignment layers route the residual stream away from it. We also show that this behaviour is seen far more in procedural reasoning, where the SLM has to think in a step-by-step fashion, than in factual recall, where the answer depends heavily on pretraining information. We quantify this with a layer-wise Criticality Score and localize the suppression to a narrow band of late layers. Building on this, we propose Latent Truth Decoding (LTD), a training-free intervention that decodes from the model’s epistemic peak rather than its final layer.