The rapid growth of scientific literature has made automated verification of scientific claims increasingly important, with Retrieval-Augmented Generation (RAG) systems emerging as a promising solution. However, their multi-stage architecture raises concerns about confirmation bias, where systems preferentially retrieve and generate evidence that supports a given claim. In this paper, we present CoBiT (Confirmation Bias Test), the first benchmark for systematically evaluating confirmation bias in RAG systems, consisting of 707 minimally perturbed claim–counterclaim pairs. We propose a unified framework with stage-specific metrics to quantify bias in both re-ranking and generation, including Ranking Bias Magnitude (RBM), Confirmation Bias Rate (CBR), and consistency measures. Our experiments on OpenScholar,ScholarQA, and LLM baselines reveal that confirmation bias arises at different stages of the RAG pipeline and can be amplified by re-ranking and generation. In particular, SciRAG achieves the lowest generation bias (CBR = 0.13), it exhibits higher bias in re-ranking, and OpenScholar shows the greatest inconsistency between claims and counterclaims. These findings highlight the need for fine-grained evaluation of epistemic risks in RAG systems and provide practical tools for diagnosing confirmation bias.

Code and Data https://github.com/Presence76/Confirmation_bias_evaluation

Peer review is central to scientific quality, yet reliance on simple heuristics, namely *lazy thinking* and non-*specific* critiques, has threatened review quality. Prior work frames *lazy thinking* detection as single-label classification and stops at detection, yet review segments often exhibit multiple co-occurring issues, and reviewers benefit more from actionable, guideline-aware feedback than from labels alone. We further show that off-the-shelf LLMs prompted for feedback frequently rewrite the entire review or address the authors rather than the reviewer, motivating an inference-time approach. We introduce an LLM-driven framework that decomposes reviews into argumentative segments, identifies issues violating ACL Rolling Review (ARR) guidelines, and generates targeted feedback using issue-specific templates refined by a novel iterative, reranking-based generation algorithm. In a controlled rewriting study, our feedback reduces guideline violations by up to 92.4%. We also release LazyReviewPlus, the *first* multi-label dataset of 1,309 sentences annotated for detecting *lazy thinking* and lack of *specificity*.¹

¹ Code and Data: https://github.com/UKPLab/emnlp2026-reviewfeedbackagent