This paper investigates the relationship between generation-time uncertainty and output reliability in probabilistic Large Language Model (LLM) decoding. Specifically, we assess the correlation between several factuality, quality and certainty metrics to infer existing patterns across state-of-the-art decoding strategies, diverse tasks and instruction-tuned models. We also propose a novel, image processing-inspired decoding approach, Otsu Thresholding, that performs adaptive, parameter-free threshold selection directly from the next-token predictive confidence distribution. The proposed Otsu-based variants achieve competitive quality and factuality across tasks and models, while exhibiting concentrated predictive distributions comparable to strong truncation-based baselines. Furthermore, we employ a selective filtering setup to analyze the impact of removing highly uncertain instances on the quality and factuality metrics, ultimately revealing that filtering based on averaged token-level uncertainty metrics can be practically useful for identifying less reliable generations, with filtering frequently improving the quality or factuality of the retained outputs.

While large, biomedical documents with complex terminology are in need of being understood more easily and efficiently, summarizing this kind of content can be problematic, as Large Language Models (LLMs) aren’t always trustworthy. Considering the importance of comprehending Cardiovascular Diseases, we study in depth the ability of different state-of-the-art biomedical LLMs to generate factual and certain summaries in this topic, and examine which generation choices can influence their trustworthiness. To that end, besides using factuality metrics, we employ techniques for token-level uncertainty estimation, an area that has received little attention from the scientific community. Our results reveal dissimilarities between LLMs and generation methods, and highlight connections between factuality and uncertainty metrics, thereby laying the groundwork for further investigation in the area.