COSE: Confidence-Orchestrated Self-Evolution for Effective LLM Reasoning

Bowen Wei, Nan Wang, Yuqing Zhou, Jinhao Pan, Ziwei Zhu


Abstract
Self-evolving large language models (LLMs) learn by generating their own training tasks and solutions, reducing reliance on human-curated supervision. However, in many reasoning domains, the model must also validate generated tasks and judge generated answers to obtain training signals. This creates a training-signal challenge: erroneous self-judgments become erroneous gradient updates. Existing approaches either rely on external verifiers, which limits generality, or treat noisy self-generated feedback as supervision. We propose COSE (Confidence-Orchestrated Self-Evolution), which uses the LLM’s intrinsic confidence as a lightweight uncertainty signal to modulate learning. COSE introduces confidence-weighted PPO updates and confidence-prioritized replay. Across 19 held-out benchmarks and four Qwen/Llama backbones (0.6B–4B), COSE consistently improves over base models and achieves the best average performance in general reasoning and mathematics, while remaining competitive on code. Code and data are available at https://github.com/weibowen555/COSE_.
Anthology ID:
2026.emnlp-main.2074
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
47229–47249
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2074/
DOI:
Bibkey:
Cite (ACL):
Bowen Wei, Nan Wang, Yuqing Zhou, Jinhao Pan, and Ziwei Zhu. 2026. COSE: Confidence-Orchestrated Self-Evolution for Effective LLM Reasoning. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 47229–47249, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
COSE: Confidence-Orchestrated Self-Evolution for Effective LLM Reasoning (Wei et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2074.pdf