Contemporary advancements in language model reasoning typically require computationally intensive reinforcement learning (RL) and massive datasets, creating barriers for resource-constrained teams. In this work, we demonstrate that high-quality, iterative training on minimal data can rival modern RL approaches. We introduce a resource-efficient framework that combines Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT) with selective guidance from larger models, iteratively refining solutions through a “reflect, rewrite, repeat” cycle (R³). Using Qwen 2.5 7B and Qwen 2.5 Math 7B as base models, our method shows meaningful performance improvements across arithmetic, symbolic and cognitive reasoning benchmarks—including GSM8K (83.1% → 88.6%), AIME’25@10 (20.0% → 30.0%) and LastLetterConcat (40.7% → 53.3%) problems. The model-agnostic nature of our R³ framework is further demonstrated through substantial improvements when applied to Mistral and LLaMA-based models. Remarkably, these gains are achieved using mere 700 basic arithmetic training samples, in stark contrast to the hundreds of thousands of examples typically required by RL-based systems. Our results suggest that reasoning improvements need not strictly depend on large-scale data. By emphasizing strategically curated training grounded in foundational principles, we achieve competitive generalization with minimal resource overhead. Our R³ pipeline also generates high-quality SFT data with high-fidelity reasoning traces as byproduct, further enabling scalable and annotation-free fine-tuning. Code is available.[https://github.com/aws-samples/sample-for-reflect-rewrite-repeat]

Multi-agent debate improves LLM reasoning, yet agreement is not evidence of correctness. When agents converge on a wrong answer through social reinforcement, consensus stopping commits that error to automated action with no recourse. We introduce Conformal Social Choice, a post-hoc layer that converts debate outputs into calibrated act-versus-escalate decisions. Verbalized probability distributions from heterogeneous agents are aggregated via a linear opinion pool and calibrated with split conformal prediction, yielding fixed-round marginal coverage: at any pre-specified round, the correct answer is included with probability ≥1-𝛼, without assumptions on individual model calibration. A hierarchical action policy maps singletons to autonomous action and larger sets to human escalation. Across eight MMLU-Pro domains and two additional benchmarks, fixed-round coverage stays close to target. The key finding is not that debate becomes more accurate, but that the conformal layer makes its failures actionable: at the final round, 81.9% of wrong-consensus cases are flagged by non-singleton sets at 𝛼=0.05. Separately, among cases that reach a singleton before the final round, accuracy reaches 90.0–96.8% on MMLU-Pro (up to 22.1pp above consensus stopping): a selection effect, not a reasoning improvement. Simple probability- and entropy-threshold rules attain comparable interception; conformal’s distinct value is a finite-sample fixed-round coverage guarantee together with explicit candidate sets for escalation. Adaptive first-singleton stopping is evaluated empirically and not covered by this guarantee.

Large language models can be confidently wrong, making uncertainty estimation especially important in black-box settings where token probabilities are unavailable. Existing post-hoc methods estimate uncertainty from output dispersion, treating sampled responses largely as exchangeable. We propose structural uncertainty, an observable signal derived from self-preference-induced ranking stability: given multiple sampled candidates, we elicit sparse pairwise self-preferences, aggregate them into ranking distributions via Bradley–Terry modeling with PageRank, and decompose uncertainty into two complementary entropy-based components—across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, self-preference-derived structural signals can provide information complementary to output dispersion, and their combination improves selective prediction on several reasoning and knowledge tasks. The two components relate differently to accuracy: on math reasoning, within-trial ambiguity can be positively correlated with correctness—consistent with settings where multiple plausible candidates remain competitive under self-preference ranking—whereas on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime where preference-based uncertainty is uninformative. Structural uncertainty is not a universal replacement for dispersion but a regime-sensitive complementary signal.