Latent Visual Reasoning (LVR) unfolds reasoning in a continuous visual embedding space, markedly improving Vision-Language Models (VLMs) on vision-intensive tasks. However, our experiments reveal a hidden cost: LVR inevitably undermines VLMs’ strong textual reasoning ability, causing substantial degradation on logic-intensive tasks. This motivates us to unify both reasoning modes within a single VLM that adaptively selects the appropriate one. We propose MoLaT (Mixture of Latent and Textual Reasoning), a novel method that trains VLMs via reinforcement learning (RL) to select LVR and text-only reasoning adaptively. MoLaT is trained in two stages. Supervised fine-tuning endows the VLM with a dual-mode capability using mixed CoT data. Then, MoLaT introduces two complementary designs: 1) The mode-adaptive advantage decouples the mode-selection signal from per-rollout correctness reward at the token level: mode-selection tokens receive dedicated cross-mode gradients, while content tokens retain the group-relative advantage. 2) The entropy-modulated advantage weight uses sequence-level entropy as a difficulty proxy, modulating signal strength to prevent premature mode commitment. Experiments confirm that MoLaT improves both vision-intensive and logic-intensive abilities, validating adaptive mode selection as key to general-purpose reasoning VLMs.

Reinforcement learning with verifiable rewards (RLVR) has recently demonstrated substantial gains in reasoning capabilities of vision–language models (VLMs). However, existing RLVR paradigms rely on deterministic rewards that implicitly penalize uncertainty expression and abstention, causing VLMs to overproduce speculative answers and degrading reliability. Through quantitative analysis, we find that vanilla RLVR, while effective in improving reasoning accuracy, significantly impairs VLMs’ ability to appropriately abstain from unanswerable queries. To systematically address this, we introduce Reliable-V-Bench, a new benchmark designed to evaluate VLMs’ abstaining capacity on challenging and realistic unanswerable visual questions. We further propose Reliable-V-RL, a reliability-aware RL framework that incorporates unanswerable and ambiguous visual questions into training, guided by a fine-grained, rubric-based reward mechanism that encourages VLMs to jointly optimize reasoning accuracy and reliable judgment. Experimental results demonstrate that Reliable-V-RL substantially improves the reliability and abstention capability of VLMs without sacrificing performance on standard multimodal benchmarks.