Learning Solvability Boundaries in LLMs: Improving Reasoning, Detecting Unsolvability, and Calibrating Refusal

Dengyun Peng, Qiguang Chen, Bofei Liu, Jiannan Guan, Libo Qin, Zheng Yan, Jinhao Liu, Jianshu Zhang, Wanxiang Che


Abstract
Reasoning alignment usually assumes each query has a determinate answer, rewarding models for reaching it and penalizing or ignoring failures. This assumption breaks on flawed queries with missing information or contradictory constraints, causing models to hallucinate invalid answers or over-refuse difficult valid tasks. We study this failure mode as solvability boundary learning: separating input-side answerability defects from model-side inability on well-posed tasks. To address this, we introduce UnsolvableQA, a verifier-backed multi-domain benchmark built with deterministic checks and audited math verification, and UnsolvableRL, an alignment framework that decouples reasoning, unsolvability detection, and capability-calibrated refusal. UnsolvableRL yields over 85% unsolvability detection on Qwen3-4B while improving solvable accuracy from 43.4% to 69.4%, with a +35.4-point solvable-accuracy gain on Llama3-8B. Diagnostic tuning further distinguishes missing-information defects from constraint conflicts and transfers to interactive clarification, improving Detail Recovery from 47.8% to 60.7% for Llama3-8B. Ablations reveal Capability Collapse: refusal penalties degrade detection without unsolvable data, but act as a rigor regularizer when such data are included.
Anthology ID:
2026.findings-emnlp.815
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
17250–17271
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.815/
DOI:
10.18653/2026.findings-emnlp.815
Bibkey:
Cite (ACL):
Dengyun Peng, Qiguang Chen, Bofei Liu, Jiannan Guan, Libo Qin, Zheng Yan, Jinhao Liu, Jianshu Zhang, and Wanxiang Che. 2026. Learning Solvability Boundaries in LLMs: Improving Reasoning, Detecting Unsolvability, and Calibrating Refusal. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 17250–17271, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Learning Solvability Boundaries in LLMs: Improving Reasoning, Detecting Unsolvability, and Calibrating Refusal (Peng et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.815.pdf