Credal Ensembles for Uncertainty Quantification in LLMs

Vani Kanjirangat, Oscar William Lithgow-Serrano, Alessandro Antonucci

Abstract
We consider the problem of uncertainty quantification in LLMs from a black-box perspective. To effectively disentangle aleatoric and epistemic uncertainty, we use an ensemble method that produces responses paired with so-called credal sets of distributions. We combine different ensembling techniques, based on temperature variation, multiple models, paraphrasing, and permutations, with semantic-similarity and verbalized elicitation methods, resulting in new, computationally cheap measures of uncertainty. A benchmark on closed-ended (BoolQ and CommonsenseQA) and open-ended (AmbigQA) questions shows good correlation with predictive accuracy. Notably, when considering ambiguous tasks, our analysis reveals a tendency of LLMs to focus on a single interpretation, an important flaw in current LLM-based solutions.
Anthology ID:
2026.uncertainlp-main.14
Volume:
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Wilker Aziz, Jonathan Berant, Bryan Eikema, Evgenia Ilia, Marie-Catherine de Marneffe, Barbara Plank, Artem Shelmanov, Swabha Swayamdipta, Jörg Tiedemann, Artem Vazhentsev, Raúl Vázquez, Chrysoula Zerva
Venues:
UncertaiNLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
235–244
Language:
URL:
https://aclanthology.org/2026.uncertainlp-main.14/
DOI:
Bibkey:
Cite (ACL):
Vani Kanjirangat, Oscar William Lithgow-Serrano, and Alessandro Antonucci. 2026. Credal Ensembles for Uncertainty Quantification in LLMs. In The Proceedings of the Third Workshop on Uncertainty-Aware NLP, pages 235–244, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Credal Ensembles for Uncertainty Quantification in LLMs (Kanjirangat et al., UncertaiNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.uncertainlp-main.14.pdf
PDF Cite Search Fix data