Shinan Zhang

Author directory

Other people with similar names: Shinan Zhang


2026

The LLM-as-a-Judge paradigm has emerged as a scalable alternative to human evaluation. However, single-model judges are limited by their inherent model biases, while multi-agent evaluation protocols that mitigate this through diverse deliberation are prohibitively expensive at inference time. To this end, we propose JudgePanel, which equips a compact Judge model with multi-agent Panel deliberation capability. Specifically, we first train on panel deliberation traces from an ensemble of strong evaluators, capturing structured patterns of discussion, disagreement, and resolution. To further improve judgment quality beyond SFT, we introduce AdaReward, an adaptive multi-reward RL algorithm that dynamically rebalances reward component weights as different objectives saturate at different rates during RL training. For practical deployment, we further design a lightweight domain specialization module for rapid adaptation to new evaluation domains with few hundred labeled samples. As a result, (i) Novel: the first framework to equip a single compact judge with multi-agent panel deliberation capability at single-model inference cost; (ii) Effective Reliable: JudgePanel with a 14B backbone outperforms judge-specialized models up to 70B across four evaluation benchmarks, demonstrates strong position consistency, and rapidly specializes to new domains with few hundred samples.
Retrieving relevant pages from visually rich documents, with evidence in charts, tables, and figures rather than body text, remains difficult: a single-pass query against a fixed index misses content in visual elements, and reordering the same candidates cannot recover a page never retrieved. We propose Reflect and Retrieve, an agentic retrieval approach centered on a Reflection Agent. After an initial page-level retrieval, the agent uses a multimodal large language model to visually reason over the retrieved page images, judge whether the evidence is sufficient, and, when gaps remain, an orchestrator issues targeted follow-up retrievals to recover the missing pages. Evaluating on two benchmarks, MMDocIR and ViDoRe V3, the Reflection Agent delivers the dominant agentic gain: it is the only configuration to beat single-step retrieval for every reasoning model on both, improving Recall@N from 0.628 to 0.663 on MMDocIR and 0.566 to 0.644 on ViDoRe V3. The improvement comes from recovering relevant pages absent in initial retrieval. Our approach indicates a clear design order: selecting an optimal embedding approach for the retrieval tool is the first step, after which the Reflection Agent contributes more value than a query decomposition agent, positioning visual reflection as the key component for document search.