Extreme low-bit quantization-aware training (QAT) is often unstable due to synchronized bin transitions, which can trigger catastrophic loss spikes. While prior works rely on scale-side controls to mitigate this fragility, we demonstrate that restricting pivot movement alone is insufficient. Instead, we identify a critical weight-side state governing low-bit QAT stability: pivot-proximal residence, where latent weights persistently linger near quantization pivots. We formalize this duration as dwell time and show that long dwell times accumulate a large, vulnerable population of weights prone to synchronized mass flips. To resolve this vulnerability, we propose Pivot Gradient Scaling (PGS) as a principled modification to the Straight-Through Estimator (STE) that leaves the forward pass intact and selectively amplifies latent-weight gradients within a localized pivot band. Crucially, we provide a mathematical guarantee proving that this sign-preserving amplification increases pivot-proximal mobility and systematically shortens expected dwell time under both SGD and Adam. Exhaustive empirical evaluations across Llama 3.2 and Qwen3 model families under full-training regimes up to 30 billion tokens demonstrate that PGS structurally stabilizes optimization, attenuates transition bursts, and delivers substantial downstream accuracy gains.