Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning using outcome-level supervision, but this supervision is temporally coarse: it indicates whether a trajectory ends with a correct answer, without revealing when the reasoning becomes reliable or silently fails. Existing verbalized-confidence methods typically calibrate a single confidence score after the final answer, leaving intermediate uncertainty unobserved. We propose SCOPE (Stepwise Calibration with Outcome-grounded Prefix Evidence), which trains reasoning models to emit confidence after each reasoning step. SCOPE interprets step confidence as the model’s prefix-conditioned belief in the gold answer: how likely the gold answer is under the reasoning prefix generated so far. During training, SCOPE constructs a soft target by scoring the gold answer conditioned on each reasoning prefix under the current policy. It then distills this signal with a two-stage masked RL objective: the first stage stabilizes the output format and final-answer accuracy, while the second stage performs calibration-only masked updates by applying stepwise and final-confidence rewards only to the numerical tokens inside the emitted confidence spans. Experiments on mathematical reasoning benchmarks show that SCOPE improves stepwise and final-confidence calibration while preserving final-answer reasoning performance. We further show that the resulting confidence signals support downstream inference-time control, enabling more reliable Learn-Then-Test risk control and stronger confidence-weighted aggregation.

Fine-tuning large language models (LLMs) faces significant memory challenges due to the high cost of back-propagation. MeZO addresses this using zeroth-order (ZO) optimization, matching memory usage to inference but suffering from slow convergence due to varying curvatures across model parameters. To overcome this limitation, We propose HELENE, a scalable and memory-efficient optimizer that integrates annealed A-GNB gradients with diagonal Hessian estimation and layer-wise clipping as a second-order pre-conditioner. HELENE provably accelerates and stabilizes convergence by reducing dependence on total parameter space and scaling with the largest layer dimension. Experiments on RoBERTa-large and OPT-1.3B show up to a 20× speedup over MeZO with an average accuracy improvement of 1.5%. HELENE supports full and parameter-efficient fine-tuning, outperforming several state-of-the-art optimizers.