Extreme low-bit quantization-aware training (QAT) is often unstable due to synchronized bin transitions, which can trigger catastrophic loss spikes. While prior works rely on scale-side controls to mitigate this fragility, we demonstrate that restricting pivot movement alone is insufficient. Instead, we identify a critical weight-side state governing low-bit QAT stability: pivot-proximal residence, where latent weights persistently linger near quantization pivots. We formalize this duration as dwell time and show that long dwell times accumulate a large, vulnerable population of weights prone to synchronized mass flips. To resolve this vulnerability, we propose Pivot Gradient Scaling (PGS) as a principled modification to the Straight-Through Estimator (STE) that leaves the forward pass intact and selectively amplifies latent-weight gradients within a localized pivot band. Crucially, we provide a mathematical guarantee proving that this sign-preserving amplification increases pivot-proximal mobility and systematically shortens expected dwell time under both SGD and Adam. Exhaustive empirical evaluations across Llama 3.2 and Qwen3 model families under full-training regimes up to 30 billion tokens demonstrate that PGS structurally stabilizes optimization, attenuates transition bursts, and delivers substantial downstream accuracy gains.

Transformer language models such as GPT-2 are difficult to quantize because of outliers in the activations leading to a large quantization error. To adapt to the error, one must use quantization-aware training, which entails a fine-tuning process based on the dataset and the training pipeline identical to those for the original model. Pretrained language models, however, often do not grant access to their datasets and training pipelines, forcing us to rely on arbitrary ones for fine-tuning. In that case, it is observed that quantization-aware training overfits the model to the fine-tuning data. To this end introduced is a quantization adapter (Quadapter), a small set of parameters that are learned to make activations quantization-friendly by scaling them channel-wise.For quantization without overfitting, we introduce a quantization adapter (Quadapter), a small set of parameters that are learned to make activations quantization-friendly by scaling them channel-wise. It keeps the model parameters unchanged. By applying our method to the challenging task of quantizing GPT-2, we demonstrate that it effectively prevents the overfitting and improves the quantization performance.