A retrieval-augmented language model returns the correct free-cash-flow figure from Microsoft’s 10-K filing. Replace the filing with a topical distractor containing no figure, and the model returns the same number: the document was never needed. We introduce RAG-CF, a training-free black-box probe that issues a per-query verdict on whether an apparently correct RAG answer is source-independent, meaning produced without the retrieved documents. The probe perturbs document context under four counterfactual operators, regenerates the answer, and converts the aggregated mismatch into a split-conformal P-value with a finite-sample Type-I bound. Across 60 configurations spanning five benchmarks, four generators, and three retrievers, the FPR-adjusted flagged rate, (raw − α)/(1 − α), ranges from 13% to 56% (raw 21% to 60%); the instrument’s FPR is domain-dependent. A held-out no-retrieval label set confirms: 78% of flagged answers remain correct without documents, against 22% of unflagged; the gap is preserved under LLM-judge equivalence. Source-independence does not decrease with retriever quality. Accuracy alone conflates grounded and parametric success; reporting the source-independence rate alongside it would make the distinction visible on every future RAG benchmark.

The EU Artificial Intelligence Act now requires providers to mark generated text so its origin can be checked. In regulated industries the duty is sharper: a marker must never flag third-party writing, survive routine review edits, and work on outputs often a sentence long. We call these the regulated-provenance triad and prove that no bounded-distortion watermark perturbing a model’s token distribution can satisfy all three below a critical length, the provenance horizon. The proof combines the Le Cam two-point method with a strong data-processing inequality for the edit channel; a matching achievability bound for the green-list family under a memoryless resampling channel makes the scaling order-tight. The barrier rises at negligible-distortion budgets, not at deployed ones. The contraction factor entering the bound is a proven upper bound for the memoryless resampling channel and a calibrated estimate for realistic paraphrase, under which we estimate most sentence-length regulated outputs fall below the horizon. Relocating provenance to a tamper-evident log read through a conformally calibrated retrieval test sidesteps the barrier: under exchangeable calibration its false-positive rate is at or below the target at every length. Coverage carries no guarantee and degrades under an adaptive editor.

LLM-as-judge leaderboards now decide what gets trained, deployed, and funded, yet years of bias patching have not asked whether the goals those patches pursue are jointly achievable. We show they are not. Casting LLM-as-judge evaluation as a social welfare functional over score profiles indexed by judge, criterion, and alternative, we prove that no aggregation rule can simultaneously satisfy four fairness axioms (Universal Domain, Weak Pareto, Independence of Irrelevant Alternatives [IIA], and Judge-Anonymity) in the ordinal regime and, among scale-invariant aggregators, in the cardinal regime. A fifth axiom, Position-Invariance, captures upstream pipeline bias and is securable independently. The impossibility admits a constructive counterpart: a four-face achievability frontier with one witness per dropped axiom. The Borda-Conformal Hybrid (BCH) occupies the dropped-IIA face, satisfies the other four axioms, and admits a structural conformal bound relating its IIA-violation probability to the calibration-set gap distribution. A 200-prompt audit on MT-Bench, AlpacaEval 2, and Arena-Hard finds IIA-violation rates from 18.5% to 38.0% across the seven aggregators with measurable IIA; a 50-prompt human-jury comparator confirms a non-zero human IIA-violation baseline consistent with known ranking pathologies in the choice-experiment literature. An audit toolkit and reporting protocol accompany the paper.

Conformal prediction supplies finite-sample coverage guarantees for individual language-model responses. We show that per-step conformal calibration need not preserve this guarantee along a multi-step agent trajectory: at a nominal miscoverage level of 10%, per-step split conformal delivers 83.5% trajectory score-threshold miscoverage on a fifteen-step web-navigation agent, and we prove lower bounds in three regimes of agent dynamics: i.i.d. resampling, memoryless covariate shift, and state-dependent Markov transitions. The i.i.d. regime alone accounts for the bulk of the empirical gap; the third regime, novel to this work, gives, to our knowledge, the first separation of the trajectory union event from i.i.d. resampling under marginal-stationary, non-trivial Markov dynamics, a setting in which prior marginal-drift analyses are silent. The separation persists under exact score stationarity and, for non-degenerate score laws, holds at any calibration sample size up to an O(T/sqrt(n)) finite-sample slack. A polynomial alpha-spending schedule restores end-to-end coverage to within 1.1 points of nominal across five agentic-NLP benchmarks and, by direct substitution, lifts process-reward-model step verifiers – which on their own miscover trajectories at 31.5-62.5% – to 9.6-10.6%, near the nominal target.