Martin Hohenadler
Author directory2026
Two Lawyers, Three Opinions: Can Output-Based LLM Uncertainty Find the Cases Where Experts Disagree?
Laura Braun | Matthias Aßenmacher | Martin Hohenadler | Goeran Kauermann
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Laura Braun | Matthias Aßenmacher | Martin Hohenadler | Goeran Kauermann
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Legal natural language processing (NLP) tasks often lack a single objective ground truth, as reflected in label variation across experts’ annotations. We test whether two uncertainty signals based on outputs of large language models (LLMs), sampling entropy and verbalized probability, can identify posts where legal experts disagree. On 1,564 German social media posts annotated by three legal experts for punishability under the German Criminal Code, we evaluate four open-weight LLMs, with reasoning either enabled or disabled, and observe a drop in F1 score on contested posts. Sampling distributions collapse to near-deterministic judgments in most conditions, while verbalized probabilities track expert disagreement in aggregate and support deferral of model errors. Neither signal, however, reliably identifies individual contested posts: both detect the model’s own errors (AUROC up to 0.83) better than expert disagreement (up to 0.76), and which signal is more informative depends strongly on model and activation of reasoning mode.