Low-bit activation-weight quantization is crucial for efficient large language model inference but remains fragile because activation quantization errors propagate across Transformer layers. We propose Activation Residual Hessian Quantization (ARHQ), a post-training residual-aware low-rank correction framework that directly targets this propagation. ARHQ measures the residuals produced by the deployed activation quantizer, builds an input-side residual Hessian, and identifies the weight directions that most amplify these residuals. This yields a closed-form truncated-SVD solution: residual-sensitive components are placed in a high-precision low-rank branch, while the remaining weights stay in the low-bit path. ARHQ requires no backpropagation, supports LoRA-style deployment, and complements activation smoothing. Across LLaMA-family models and Qwen3-4B-Thinking, ARHQ improves W4A4 reconstruction, reduces WikiText2 perplexity over matched-rank SVDQuant, preserves downstream reasoning and coding accuracy, and achieves up to 3.21× prefill speedup over BF16 (3.11× over AWQ W4A16) with over 60% memory saving.

Recent advancements in Large Language Models (LLMs) have demonstrated enhanced reasoning capabilities, evolving from Chain-of-Thought (CoT) prompting to advanced, product-oriented solutions like OpenAI o1. During our re-implementation of this model, we noticed that in multimodal tasks requiring visual input (e.g., geometry problems), Multimodal LLMs (MLLMs) struggle to maintain focus on the visual information, in other words, MLLMs suffer from a gradual decline in attention to visual information as reasoning progresses, causing text-over-relied outputs. To investigate this, we ablate image inputs during long-chain reasoning. Concretely, we truncate the reasoning process midway, then re-complete the reasoning process with the input image removed. We observe only a ~2 accuracy drop on MathVista’s test-hard subset, revealing the model’s textual outputs dominate the following reasoning process. Motivated by this, we propose Take-along Visual Conditioning (TVC), a strategy that shifts image input to critical reasoning stages and compresses redundant visual tokens via dynamic pruning. This methodology helps the model retain attention to the visual components throughout the reasoning. Our approach achieves state-of-the-art performance on average across five mathematical reasoning benchmarks (+3.4% vs previous sota), demonstrating the effectiveness of TVC in enhancing multimodal reasoning systems. The project page is available at https://sun-hailong.github.io/projects/TVC.