2026

LLM-as-judge leaderboards now decide what gets trained, deployed, and funded, yet years of bias patching have not asked whether the goals those patches pursue are jointly achievable. We show they are not. Casting LLM-as-judge evaluation as a social welfare functional over score profiles indexed by judge, criterion, and alternative, we prove that no aggregation rule can simultaneously satisfy four fairness axioms (Universal Domain, Weak Pareto, Independence of Irrelevant Alternatives [IIA], and Judge-Anonymity) in the ordinal regime and, among scale-invariant aggregators, in the cardinal regime. A fifth axiom, Position-Invariance, captures upstream pipeline bias and is securable independently. The impossibility admits a constructive counterpart: a four-face achievability frontier with one witness per dropped axiom. The Borda-Conformal Hybrid (BCH) occupies the dropped-IIA face, satisfies the other four axioms, and admits a structural conformal bound relating its IIA-violation probability to the calibration-set gap distribution. A 200-prompt audit on MT-Bench, AlpacaEval 2, and Arena-Hard finds IIA-violation rates from 18.5% to 38.0% across the seven aggregators with measurable IIA; a 50-prompt human-jury comparator confirms a non-zero human IIA-violation baseline consistent with known ranking pathologies in the choice-experiment literature. An audit toolkit and reporting protocol accompany the paper.
Conformal prediction supplies finite-sample coverage guarantees for individual language-model responses. We show that per-step conformal calibration need not preserve this guarantee along a multi-step agent trajectory: at a nominal miscoverage level of 10%, per-step split conformal delivers 83.5% trajectory score-threshold miscoverage on a fifteen-step web-navigation agent, and we prove lower bounds in three regimes of agent dynamics: i.i.d. resampling, memoryless covariate shift, and state-dependent Markov transitions. The i.i.d. regime alone accounts for the bulk of the empirical gap; the third regime, novel to this work, gives, to our knowledge, the first separation of the trajectory union event from i.i.d. resampling under marginal-stationary, non-trivial Markov dynamics, a setting in which prior marginal-drift analyses are silent. The separation persists under exact score stationarity and, for non-degenerate score laws, holds at any calibration sample size up to an O(T/sqrt(n)) finite-sample slack. A polynomial alpha-spending schedule restores end-to-end coverage to within 1.1 points of nominal across five agentic-NLP benchmarks and, by direct substitution, lifts process-reward-model step verifiers – which on their own miscover trajectories at 31.5-62.5% – to 9.6-10.6%, near the nominal target.