2026

Large vision-language models (LVLMs) remain highly sensitive to text embedded in images: a misleading word printed, painted, or pasted onto an object can override visual evidence and flip the model’s answer. Existing defenses are often tied to specific vision encoders, require training or feature-level intervention, or do not operate on the autoregressive decoding interface of modern LVLMs. We propose TypoCD, a training-free contrastive decoding method for typographic robustness. TypoCD constructs a text-localized counterfactual view by detecting visible text regions and inpainting only the typographic content, then contrasts the attacked-image and typography-removed logits during decoding. This localized contrastive direction targets the attacker’s textual signal while preserving unrelated visual content, allowing TypoCD to subtract the typography-induced logit shift without modifying model parameters or internal features. Across three open-source LVLM backbones and synthetic, real-world, and safety-oriented typographic benchmarks, TypoCD recovers attacked-image accuracy to within a few percentage points of clean-image performance without per-backbone tuning, closing a gap that often exceeds 40 percentage points under attack. When combined with a simple prompt prefix, TypoCD also substantially reduces jailbreak attack success on safety-oriented typographic attacks.