2026

Post-training with internal feedback can improve a targeted knowledge-grounded behavior while silently degrading complementary abstraction and logical-consistency capabilities. We propose a conjunctive evaluation protocol that accepts a method only when a designated target improves and complementary probes remain non-inferior (with a stricter secondary criterion requiring complementary equivalence), while also reporting retention and predictive drift. We instantiate the protocol in DION-Bench, combining two lexical/ontological probes (same-category exceptions; abstraction recovery) with a synthetic SAT-checked logical-consistency probe. On Qwen3-8B-Base, a task-specific self-judged DPO adaptation improves OntoNeg-WN by 5.27 percentage points but reduces ConceptShift by 37.50 points—despite humans reaching 82.7% and frontier LLMs 44–73% on OntoNeg—whereas mean change across five external accuracy suites is only −1.70 points. A development-tuned decoding rule partially mitigates ConceptShift regression to 12.93/14.60 points on the original/holdout splits while retaining most of the target gain. These results show that faithful post-training claims require joint reporting of gain, complementary abstraction regression, retention, drift, and equivalence—not proof that a single shared knowledge item degraded.
Multi-turn scheduling assistants must preserve constraint edits and distinguish solver evidence from language-model interpretations. We present CoSched-Cert, a dialogue protocol combining a typed scheduling language, solver-distinguishing clarification, cost-based conflict localization, verified repair synthesis, and lexical certificate auditing. Its guarantees are conditional on the encoded problem and supplied candidate interpretations; lexical support does not establish semantic entailment. SchedNL-Bench-Cert contains 1,000 synthetic Chinese/English utterances and 350 sessions. Gold-AST replay measures internal consistency against labels generated by the same engine, while author-written protocol simulations illustrate selected design choices. Across three live-parser seeds on 52 sessions, session training and registered sibling augmentation raise conditional micro target-constraint agreement from 18.0% to 83.7%. The final configuration has 100% coarse status agreement but only 75.0% conflict-member agreement, and includes a rule informed by a test-session failure. We therefore treat these results as descriptive, not as an unbiased generalization estimate. We define a stricter joint-turn criterion and report the bounds supported by the available aggregates. The contribution is a protocol for exposing inspectable solver evidence and the remaining gap between encoded feasibility and user-intent correctness.
Retrieval-augmented generation treats citation as a default proxy for reliability, yet a cited passage can support a claim and still be obsolete, superseded, contradicted, or invalid at the user’s target time. We name this failure a citation-supported lifecycle error and define lifecycle faithfulness: a claim-level criterion that requires semantic support, target-time validity, non-supersession, and provenance. We release Vera-Life-Eval, an 8,500-question benchmark with 41,203 versioned evidence pieces across six lifecycle settings (current, past, stale-only, update, conflict, and multi-hop), and Vera-Life, a low-training modular controller that builds an evidence-state graph, selects claim-evidence-time contracts, and chooses among answer, warn_stale, explain_conflict, retrieve_more, and abstain, emitting a claim-level lifecycle ledger. On a shared Qwen3-8B generator, Vera-Life raises LifeEM from .374 (best prompt-only baseline) to .623, and remains effective under timestamp-only (.584) and text-only (.504) regimes. Same-action controls, metadata isolation, and source-disjoint transfer across product, policy, scientific, and news evidence show that the gains are not an artefact of output format or Wikipedia revision cues. The next mission for RAG is not merely to cite evidence, but to know whether that evidence is still alive.
Current RAG evaluation measures answer quality under fixed evidence but cannot detect systems that ignore evidence changes or fall back to parametric knowledge. We introduce Dynamic Evidence Contracts, a behavioral evaluation paradigm that specifies expected RAG behavior under four evidence interventions (deletion, conflict, distraction, permutation), and the Intervention Compliance Score (ICS) to quantify compliance. Evaluation on five datasets reveals that all evaluated open-source RAG pipelines achieve moderate conflict detection (CDR=0.283–0.847) via passive NLI-detected refutation signals, but lack explicit conflict acknowledgment. We further present DOVE-RAG, a minimal reference instantiation showing that explicit conflict acknowledgment yields higher CDR (0.505–0.999), up to 1.6× ICS improvement, and 2.0× citation recall gains. Ablation reveals that verification without structured output reorganization yields negligible improvement, suggesting that evidence compliance requires an integrated pipeline.
When large language models serve as evaluation infrastructure, progress cannot be reduced to a single accuracy number. We study winning-argument (WA) detection as a stress test: can one system jointly improve accuracy, enforce speaker-order invariance, emit rubric-conditioned diagnostics, improve a proper scoring rule, and cut inference cost? TAPIR freezes a survey-grounded rubric into a dual-artifact protocol, distils privileged signals by mode-split self-distillation, and predicts through an anti-symmetric scalar head paired with a structured ledger. An algebraic swap identity (Winning-Position Invariance, WPI=1) holds on every test pair, and a tied-order wrapper preserves it on the deployed router. On the standard 807-pair WA split, TAPIR beats our same-backbone Qwen3-14B re-implementation of Labruna et al.’s MS-PS-MLP, 65.80% vs. 58.12% (paired McNemar p=3.41×10−4), at a lower Brier score and 39.41× faster inference per pair. Reusing the frozen protocol without retuning carries the mechanism to a second task and a second backbone, both at WPI=1. Under domain shift it only matches length-only controls, so we report contract preservation rather than superiority. The contribution is the joint contract under a paired protocol, with an explicit amortisation boundary, not a leaderboard delta.
When LLM-generated emergency advice becomes part of public-facing information infrastructure, aggregate accuracy is no longer an adequate safety target: missing a hazardous instruction and blocking a correct one impose different human costs. We study protocol-grounded verification, the post-hoc auditing of emergency instructions against authoritative guidance under asymmetric risk. We introduce EMER-GuardBench, a 69,000-example counterfactual benchmark plus a 1,000-example benign-stress set spanning 10 disaster scenarios and 6 hazard mechanisms, with instruction-group-disjoint splits and cited protocol evidence. We further propose EGIS, an evidence-conditioned, protocol-supervised verifier that jointly predicts hazard status, hazard mechanism, severity, evidence support, and counterfactual ranking. This establishes a protocol-conditioned task and protocol-supervised representation learning, not inference-time causal grounding. EGIS reaches 89.9 Harm F1 with 10.4% IID over-refusal and zero benign-stress false positives; across 364 LLM-judge operating points, none Pareto-dominates EGIS on Harm F1, over-refusal, and severity-weighted miss rate, and a 14.4×-larger Qwen3-8B+LoRA supervised baseline still trails under identical splits. Ablations show that evidence helps as training-time supervision rather than prompt decoration. Our Severity-Weighted Miss Rate exposes a tradeoff hidden by scalar safety scores: zero missed severity can be achieved by broad refusal. EMER-Guard reframes emergency LLM safety from advice generation to auditable, protocol-grounded, cost-sensitive verification.