Large Language Models (LLMs) show strong potential in medical QA, but still suffer from hallucinations in multi-step clinical reasoning. Existing methods fail to assess confidence at intermediate steps or reliably integrate external knowledge, leading to propagated errors and misleading conclusions. Inspired by Bayesian inference logic in clinical diagnosis, we propose BELIEF, a Bayesian Evidence Leveraged InfErence Framework that enables LLMs to perform reliable medical reasoning through dynamic belief assessment and knowledge-augmented disambiguation. First, the Bayesian Belief Updating Module (BBUM) treats diagnostic hypotheses as prior probabilities and calculates posterior probabilities based on observed clinical evidence, preventing early-stage error propagation with statistical rigor. Second, the Dynamic Evidence Retrieval Module (DERM) autonomously queries external medical databases to resolve ambiguity when the posterior belief exhibits high entropy. Experiments across 7 datasets and 2 LLM backbones show that BELIEF significantly improves diagnostic accuracy, enhances interpretability, and mitigates hallucinations through explicit probabilistic updates.