Low-bit activation-weight quantization is crucial for efficient large language model inference but remains fragile because activation quantization errors propagate across Transformer layers. We propose Activation Residual Hessian Quantization (ARHQ), a post-training residual-aware low-rank correction framework that directly targets this propagation. ARHQ measures the residuals produced by the deployed activation quantizer, builds an input-side residual Hessian, and identifies the weight directions that most amplify these residuals. This yields a closed-form truncated-SVD solution: residual-sensitive components are placed in a high-precision low-rank branch, while the remaining weights stay in the low-bit path. ARHQ requires no backpropagation, supports LoRA-style deployment, and complements activation smoothing. Across LLaMA-family models and Qwen3-4B-Thinking, ARHQ improves W4A4 reconstruction, reduces WikiText2 perplexity over matched-rank SVDQuant, preserves downstream reasoning and coding accuracy, and achieves up to 3.21× prefill speedup over BF16 (3.11× over AWQ W4A16) with over 60% memory saving.