Tool-augmented visual reasoning systems, exemplified by Visual Programming (VP), decompose complex visual questions into executable programs that invoke specialized visual models as tools, offering interpretable and compositional reasoning. However, these systems face a fundamental optimization challenge: the non-differentiable pipeline and the absence of intermediate supervision prevent end-to-end gradient-based optimization of invoked tools using only task-level labels. Existing methods either optimize the Large Language Model (LLM) planner while neglecting tool quality, or rely on teacher models and additional data collection, none achieving true end-to-end differentiable optimization. We propose **eDoT**, the first **e**nd-to-end **D**ifferentiable **o**ptimization method for **T**ool-augmented visual reasoning. Our key insight is that any program’s variable dependency structure is isomorphic to a directed probabilistic graph. Applying Variable Elimination - an exact probabilistic inference technique - transforms the non-differentiable VP execution into a fully differentiable process, enabling gradient-based optimization with only task-level supervision, without additional data or teacher models. Experiments on GQA, NLVRv2 and Open Images demonstrate improvements of 3.7%, 7.6% and 20%, validating eDoT’s effectiveness across multiple visual models and cross-task generalization.