Animesh Mishra

Author directory

2026

The WMT26 shared task on automated translation quality evaluation asks participants both to build metrics (Task 2) and to submit challenge sets that probe them (Task 4). We describe one submission to each, and a single finding that connects them.

Modern machine translation (MT) metrics cannot detect a class of invisible Unicode corruption, and the reason is not that they judge it wrongly: their tokenisers delete the evidence first. Four such corruptions — a non-breaking space, a narrow non-breaking space, an ideographic space, and a leading byte-order mark — normalise to token sequences identical to the clean text. This holds under both XLM-R and mT5, the encoders behind COMET-22 and MetricX-24. The metric therefore receives the same input and returns the same score, and no amount of training can separate the pair. Perturbations that instead fragment tokenisation are caught reliably.

LIGATUR (Task 4) is the 171-item contrastive challenge set that isolates this effect for English-German (en-de) and English-Hindi (en-hi); a blind re-evaluation with held-out labels also overturns our own pilot claim that judges ignore Devanagari conjunct breaks. AEGIS (Task 2) acts on the diagnosis, combining a reference-based ensemble with deterministic hygiene penalties for exactly what the encoders cannot see, and scores all 26,885 test items. An ablation on 34,174 WMT24 judgements shows the two-family ensemble is our strongest component and that the reference-free branch we added as insurance costs accuracy where references are good, which we report as a calibration error.
Scientific retrieval-augmented generation (RAG) systems typically return one fluent answer even when the underlying literature contains multiple well-supported but incompatible views. We call this failure mode epistemic collapse: compression of structured disagreement into a single response. We present EVIRAG (Evidence-centric, Viewpoint-Inclusive RAG), a framework for scientific question answering that combines adversarial retrieval, claim-level disagreement graphs, causal disagreement attribution, multi-view synthesis, and evidence structured confidence labels. We also introduce EVIRAG-BENCH, a 1,250-query benchmark spanning five contested scientific domains. Across eight compared systems, EVIRAG substantially outperforms vanilla RAG on ContradictionRecall and dense-encoder ViewpointCoverage while reducing Consensus Collapse Score. Human evaluation with three annotators confirms markedly higher epistemic completeness for EVIRAG than for vanilla RAG (Krippendorff’s α=0.72, p<0.001, Cohen’s d=2.71). An offline precomputed graph reduces warm-query latency by over an order of magnitude. These results suggest that treating disagreement preservation as a primary design objective substantially improves how scientific RAG systems represent contested evidence in disagreement-heavy QA settings.