Large language models (LLMs) and vision-language models (VLMs) are increasingly used as optimization assistants to produce solutions, generate solver-executable programs, or both. However, current evaluations are misaligned with deployment in three ways: they (P1) fail to represent multimodal problem specifications, (P2) score outcomes only and cannot localize where failures occur along the modeling pipeline, and (P3) rarely report inference cost, obscuring reliability–cost trade-offs. We introduce Graph Optimization benchmark (GOBench), an aligned multimodal benchmark with solver-derived oracles and a four-layer diagnostic protocol that evaluates intermediate artifacts as well as end results, together with the Visual Inference Penalty (VIP) to measure multimodal overhead. Across frontier and open-weight models under paired text-only vs. T+V settings, we find that vision reliably increases inference cost, while its reliability impact is regime-dependent: frontier models often benefit from visual grounding, whereas several mid-tier/open models exhibit a Visual Paradox where vision reduces downstream executability and verification coverage. End-to-end success is frequently bottlenecked by intermediate-stage dropout; supervised fine-tuning on intermediate targets can mitigate this attrition in open models, enabling a reproducible harness for diagnosing failure modes and quantifying reliability–cost trade-offs.

Large language models make natural-language-to-MILP modeling increasingly plausible, but reliably selecting among multiple generated programs remains challenging: structurally similar errors can create false majorities, while solving every candidate integer program is costly. We propose Structural-Semantic Consistency (SSC), a best-of-K selector combining structural consistency, static quality, text alignment, and two diagnostics obtained from continuous LP relaxations. Notably, SSC does not require solving any candidate integer programs during selection. Across seven benchmarks and 618 fixed candidate pools, SSC obtains 55.7% accuracy, compared with a random baseline of 51.5%, 53.7% for a zero-shot listwise LLM judge, and 55.3% for a leave-one-benchmark-out learned ranker. The symbolic fusion reaches 53.6%, with LP diagnostics adding 2.1 percentage points. These results establish multi-signal weak verification as a competitive and interpretable baseline for candidate selection.