Nika Jacob
Author directory2026
On the Neutrality of Model Confidence in German and English Gender-Bias Benchmarks
Shalaka Satheesh | Nika Jacob | Katharina Beckh
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Shalaka Satheesh | Nika Jacob | Katharina Beckh
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Confidence, the probability a model assigns to its prediction, is commonly treated as a signal for correctness, and is central to quality monitoring and selective question answering. We ask whether this signal is neutral with respect to social stereotypes. A model’s confidence may reflect correctness well overall and yet be higher where its answers align with a stereotype and detecting this requires comparing confidence across stereotype directions. We investigate this on the gender subset of Bias Benchmark for Question Answering (BBQ) in English and in German (GG-BBQ), evaluating 22 base and instruction-tuned models from five providers. For some models, we find small but directionally consistent effects for both ambiguous and disambiguated questions, suggesting that confidence-based selection can favour stereotype-aligned content and that confidence is worth examining alongside answer-based scores in bias evaluations on multiple-choice questions.