Subjective NLP tasks such as sentiment analysis and hate speech classification often involve inherent annotator disagreement, reflecting diverse perspectives shaped by annotators’ lived experiences. Although conventional approaches resolve disagreement through majority voting or aggregation, these methods risk erasing valuable nuances and minority viewpoints. Recent embedding-based and multitask models have advanced the modeling of annotator-specific judgments, yet their robustness to annotation noise remains underexplored. In this work, we systematically investigate how state-of-the-art multi-annotator learning models perform in the presence of noisy labels, and find that embedding-based approaches are more robust than aggregation and multitask baselines on datasets with many sparse annotators. Furthermore, we propose a new methodology, Noise-Robust Annotator Embedding (NRA-Embed), which integrates Robust InfoNCE (RINCE) contrastive loss to enhance models’ robustness under noisy annotation conditions. Through extensive experiments, we demonstrate that NRA-Embed effectively captures subjective disagreement while maintaining robustness to noise, achieving performance that is comparable to state-of-the-art in clean settings while providing numerically greater robustness under noise.

Large Language Models (LLMs) are increasingly used as conversational search engines for academic literature, yet whether they judge papers on content or on authority signals has not been tested causally. We investigate authority bias: systematic preference for papers based on author prestige, venue, and citations rather than content. Holding title and abstract constant, we vary authority metadata across three counterfactual conditions (original, flipped, boosted) over eight LLMs (five open-weight and three frontier closed-weight) in an in-context, single-turn, top-1 recommendation setting. Our experiments show that authority bias is substantial and directional, varies markedly across models, and is only partially addressable through prompt-level debiasing. We further document a say-do gap: debiasing instructions suppress authority mentions far faster than authority-driven flips, so surface auditing systematically underestimates behavioral bias.

Code: https://github.com/jinaduuthman/Authority-Bias-In-Conversational-Search-Engine

Data: https://huggingface.co/datasets/uthmanjinadu/authority-bias-paper-recommendation