Self-consistency improves LLM reasoning by majority voting over multiple sampled solutions, but always exhausts the sampling budget and trusts every solution equally. Early-stopping methods cut token cost substantially via convergence monitoring. However, deciding when to stop cannot change what is voted on, so they save tokens without meaningfully improving accuracy. We introduce CASE (Confidence-Aware Self-consistency with Adaptive Early stopping), which improves both accuracy and efficiency by leveraging token-level confidence normalized at two scales: within-problem to filter low-quality solutions and weight survivors, and cross-problem to adapt the stopping threshold to problem difficulty. Across 5 models and 6 benchmarks, CASE simultaneously saves 85% of tokens and improves accuracy by +2.24pp over majority voting, outperforming every early-stopping baseline on both accuracy and efficiency. Component and role-swap analyses confirm that filtering and weighting are complementary and the two normalization scales are non-interchangeable.