Maciej Skorski
Author directory2026
Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment
Maciej Skorski
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Maciej Skorski
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Most work in computational ethics treats annotator disagreement on moral content as noise to be voted away, collapsed into *majority* vote or the more permissive *any-annotator* rule which assigns a label as soon as any annotator flags it.
We argue this uncertainty should instead be modeled and learned from.
We introduce **Moral Entropy**, a Bayesian framework that estimates a full posterior over the true label and decomposes its entropy into *aleatoric* uncertainty (irreducible disagreement about the moral content) and *epistemic* uncertainty (from insufficient or noisy annotation). This lets any heuristic consensus rule be audited against a calibrated ground truth via entropy methods such as cross-entropy/KL, Brier score, and expected calibration error.
Running this model on over 100,000 posts from three corpora spanning fifteen discourse domains, we audit the standard aggregation rules against the resulting posterior and reveal systematic differences in moral labels. Across MFTC and MFRC, the any-annotator rule yields mean false-positive rates of 10.2–19.9% and mean false-negative rates of 38.9–54.8% relative to the Bayesian consensus, while majority-vote and two-vote rules miss 63–83% of reference-positive labels.
We argue this uncertainty should instead be modeled and learned from.
We introduce **Moral Entropy**, a Bayesian framework that estimates a full posterior over the true label and decomposes its entropy into *aleatoric* uncertainty (irreducible disagreement about the moral content) and *epistemic* uncertainty (from insufficient or noisy annotation). This lets any heuristic consensus rule be audited against a calibrated ground truth via entropy methods such as cross-entropy/KL, Brier score, and expected calibration error.
Running this model on over 100,000 posts from three corpora spanning fifteen discourse domains, we audit the standard aggregation rules against the resulting posterior and reveal systematic differences in moral labels. Across MFTC and MFRC, the any-annotator rule yields mean false-positive rates of 10.2–19.9% and mean false-negative rates of 38.9–54.8% relative to the Bayesian consensus, while majority-vote and two-vote rules miss 63–83% of reference-positive labels.
2025
Beyond Human Judgment: A Bayesian Evaluation of LLMs’ Moral Values Understanding
Maciej Skorski | Alina Landowska
Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025)
Maciej Skorski | Alina Landowska
Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025)
How do Large Language Models understand moral dimensions compared to humans?This first comprehensive large-scale Bayesian evaluation of leading language models provides the answer. In contrast to prior approaches based on deterministic ground truth (obtained via majority or inclusion consensus), we obtain the labels by modelling annotators’ disagreement to capture both aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty (model domain sensitivity).We evaluated Claude Sonnet 4, DeepSeek-V3, and Llama 4 Maverick across 250K+ annotations from nearly 700 annotators in 100K+ texts spanning social networks, news, and discussion forums. Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing that AI models generally rank among the top 25% of annotators in terms of balanced accuracy, substantially better than average humans.Importantly, we find that AI produces far fewer false negatives than humans, highlighting their more sensitive moral detection capabilities.