Production Retrieval-Augmented Generation (RAG) pays retrieval latency and serving cost on every query – even when the model already knows the answer. We present CONFRAG, a confidence-guided strategy that invokes RAG only when a large language model (LLM) is unlikely to be correct, cutting external retrievals by 5-39% and tail latency by up to 800ms (P90) in a real RAG system while matching the answer quality of always-on RAG. CONFRAG is enabled by CONFQA, a lightweight fine-tuning strategy (one epoch over 3K knowledge-graph facts) that teaches an LLM to answer only when confident and otherwise say "I am unsure", cutting hallucination rates from 20-40% to 5% or below across multiple factuality benchmarks. Two choices make it work in practice: a dampening prompt that discourages overconfident hallucinations, and training on atomic facts that generalizes confidence calibration to unseen domains and question types. At serving time, LLM-QA and RAG run in parallel with early-stopping, so confident answers return immediately.