Graph learning requires jointly leveraging node semantics and graph structure, but traditional methods often suffer from label scarcity and shallow semantic utilization. Although LLMs provide strong semantic reasoning for graph learning, existing LLM-augmented pipelines face two major challenges: their semantics-only outputs can inject semantic noise that conflicts with graph topology, and they often incur substantial overhead due to LLM fine-tuning or repeated LLM calls at training/inference. We propose CoFGraph, a confidence-feedback framework that interleaves reciprocal calibration between semantic and structural views to enable reliable and efficient LLM-augmented graph learning. To avoid expensive LLM adaptation, we keep the LLM frozen and train a lightweight Interpreter as a semantic proxy, eliminating the need for LLM fine-tuning and continual test-time prompting. To further mitigate semantic noise and the lack of structural feedback, we introduce confidence-based dual-view learning with node-wise confidence estimation and dynamic role switching, allowing the more reliable view to correct the other. Experiments on four public benchmarks validate the effectiveness of CoFGraph and demonstrate strong stability across varying training label ratios, with particularly consistent gains in low-resource regimes.