Large vision-language models (LVLMs) exhibit strong multimodal reasoning capabilities, yet reliably estimating whether a generated response is correct remains a fundamental challenge. Existing confidence estimation methods commonly rely on repeated querying, self-consistency sampling, or layer-wise probing, which introduce additional inference overhead, self-preference bias, or costly full-depth analysis. In this work, we propose a trajectory-based framework for estimating LVLM response correctness from a single forward generation pass. Instead of treating internal representations as isolated layer-wise signals, we model response generation as a structured trajectory through the LVLM, capturing the evolution of visual evidence in the vision encoder, geometric transformations induced by cross-modal projection, and probability dynamics within the language decoder. Our key insight is that response correctness is encoded in the geometry and dynamics of these component-specific trajectories. Based on this observation, we derive lightweight trajectory descriptors for each major LVLM module and integrate them through a calibrated correctness estimator trained with as few as 500 labeled examples. At test time, our method requires no additional inference passes and no ground-truth labels. Extensive experiments across multiple LVLM architectures and benchmark datasets demonstrate consistent improvements over existing baselines, including up to 16.44 higher AUROC on Qwen-2B over VisWiz and 15.61 higher AUPR on LLaVA-Next-7B over ScienceQA compared to the strongest competing methods.

Recent advances in large Vision-Language Models (VLMs) have led to impressive performance on chart understanding tasks. However, existing evaluations are largely limited to clean, canonical charts and do not adequately measure robustness under realistic conditions. To address this gap, we introduce ROBUSTCHART360, a comprehensive benchmark for evaluating chart understanding robustness under realistic real-world variations. ROBUSTCHART360 spans six robustness dimensions covering textual and cross-modal interference, rendering and display degradation, accessibility-oriented transformations, and visual style variation. Using this benchmark, we conduct a large-scale evaluation of 14 proprietary and open-source VLMs across 2,505 chart instances and multiple robustness conditions designed to simulate practical deployment scenarios. Our analysis on 35,070 model responses reveals that current VLMs are highly sensitive to seemingly minor variations such as low-resolution rendering, and mobile layouts, with performance dropping by as much as 33.9% under certain conditions, such as modality gap. We further find that models often over-rely on textual priors when visual and textual signals conflict, indicating weak visual grounding despite strong performance on standard benchmarks. Our findings highlight a substantial gap between benchmark accuracy and real-world robustness, motivating the development of more reliable and visually grounded multimodal systems for chart understanding. We will release our code and dataset at <redacted>.