Contemporary advancements in language model reasoning typically require computationally intensive reinforcement learning (RL) and massive datasets, creating barriers for resource-constrained teams. In this work, we demonstrate that high-quality, iterative training on minimal data can rival modern RL approaches. We introduce a resource-efficient framework that combines Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT) with selective guidance from larger models, iteratively refining solutions through a “reflect, rewrite, repeat” cycle (R³). Using Qwen 2.5 7B and Qwen 2.5 Math 7B as base models, our method shows meaningful performance improvements across arithmetic, symbolic and cognitive reasoning benchmarks—including GSM8K (83.1% → 88.6%), AIME’25@10 (20.0% → 30.0%) and LastLetterConcat (40.7% → 53.3%) problems. The model-agnostic nature of our R³ framework is further demonstrated through substantial improvements when applied to Mistral and LLaMA-based models. Remarkably, these gains are achieved using mere 700 basic arithmetic training samples, in stark contrast to the hundreds of thousands of examples typically required by RL-based systems. Our results suggest that reasoning improvements need not strictly depend on large-scale data. By emphasizing strategically curated training grounded in foundational principles, we achieve competitive generalization with minimal resource overhead. Our R³ pipeline also generates high-quality SFT data with high-fidelity reasoning traces as byproduct, further enabling scalable and annotation-free fine-tuning. Code is available.[https://github.com/aws-samples/sample-for-reflect-rewrite-repeat]

Large language models can be confidently wrong, making uncertainty estimation especially important in black-box settings where token probabilities are unavailable. Existing post-hoc methods estimate uncertainty from output dispersion, treating sampled responses largely as exchangeable. We propose structural uncertainty, an observable signal derived from self-preference-induced ranking stability: given multiple sampled candidates, we elicit sparse pairwise self-preferences, aggregate them into ranking distributions via Bradley–Terry modeling with PageRank, and decompose uncertainty into two complementary entropy-based components—across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, self-preference-derived structural signals can provide information complementary to output dispersion, and their combination improves selective prediction on several reasoning and knowledge tasks. The two components relate differently to accuracy: on math reasoning, within-trial ambiguity can be positively correlated with correctness—consistent with settings where multiple plausible candidates remain competitive under self-preference ranking—whereas on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime where preference-based uncertainty is uninformative. Structural uncertainty is not a universal replacement for dispersion but a regime-sensitive complementary signal.