Guodong Guo

Author directory

2026

KV cache quantization is well studied for autoregressive language models, where cached prefix states are exact under causal attention and quantization is mainly a cache reconstruction problem. Diffusion language models (dLLMs) break this assumption. Fast dLLM inference reuses KV states across denoising steps, but bidirectional token representations can change as masked tokens are revealed, so the reused cache may already differ from full recomputation before quantization is applied. We study KV cache quantization under this cache-reuse error. We adapt KIVI and KVQuant to Fast-dLLM-style prefix and dual cache reuse, and introduce TraceKV and KProtect. TraceKV reconstructs feasible dLLM reveal trajectories from prompt-answer pairs, providing calibration states that match iterative masked decoding. KProtect uses these trajectories to choose per-layer K-cache entries that remain in high precision according to output- and attention-level behavior metrics, while applying standard low-bit value quantization. Across LLaDA and Dream evaluations on GSM8K, HumanEval, MATH, and MBPP, KProtect achieves the highest average score among quantized methods across the reported 4-bit and 2-bit prefix/dual cache settings.
Object hallucination in Multimodal Large Language Models (MLLMs) often appears in two forms: omitting objects that are present in the image and generating objects that are absent. Existing training-free mitigation methods usually apply a uniform intervention to both cases. We find that this strategy leads to an imbalanced effect: it can reduce some omission errors, but often introduces new fabrications. To understand this behavior, we analyze spatial attention patterns and layer-wise attention dynamics during generation. Our results suggest that omissions often occur when the model attends to the relevant visual regions but gradually loses visual confidence during decoding, whereas fabrications are more closely related to deep-layer semantic priors overriding visual evidence. Based on this observation, we propose Visual Potential Field Calibration (VPFC), a training-free hidden-state calibration method. VPFC locates an intervention region using the spatial centroid of high-confidence visual tokens and applies a lightweight hidden-state update to adjust the model’s generative confidence. Experiments on multiple hallucination benchmarks show that VPFC consistently improves over existing training-free methods while maintaining inference efficiency close to standard greedy decoding.