Tool-augmented medical agents promise clinical control and interpretability, yet their specialized outputs are often invoked broadly and compressed into text. It remains unclear whether failures arise from this textual interface or low-fidelity upstream tools. We study this question through RouteVerify, a transparent diagnostic sandbox combining a lightweight linear router, compact evidence cards, and verifier-gated multimodal reinspection. On matched and held-out ChestAgentBench splits, RouteVerify improves answer accuracy by about seven points over naive always-call-tools execution while roughly halving token use. However, it remains about two points below a direct VLM-only baseline and uses nearly twice the tokens. Oracle probes clarify the bottleneck: replacing predicted evidence with benchmark-side gold explanations raises the same downstream text reasoner above 95 percent accuracy, or above 91 percent after removing explicit answer cues. Thus, text cards can express sufficient evidence on this benchmark; the dominant practical limitation is upstream tool quality. An external JSRT/SCR audit supports this diagnosis, revealing limited nodule-classification discrimination and structure-dependent segmentation quality. The remaining raw-versus-sanitized oracle gap also indicates sensitivity to human-authored cues. We present RouteVerify as an empirical boundary study of where current tool-augmented medical agents fail.