Machine translation shared tasks publish several automatic metrics and rank systems on one of them, assuming the choice is inconsequential. We test this assumption on the WMT 2025 and WMT 2026 Low-Resource Indic Language Translation tasks, where pre-trained multilingual encoders are unavailable for several target scripts and languages. As no human gold standard evaluation exists at scale, we adapt the Quantified Reproducibility Assessment (QRA) framework across 20 WMT26 and 14 WMT25 directions, to measure agreement between six automatic metrics and two LLM judges as independent measuring instruments. We show that QRA’s Type I measure fails under this repurposing when instruments sit on different scales, and suggest a rank-based repair. Using a script-controlled natural experiment (Manipuri evaluated in both Bengali and Meitei Mayek) and a null-output validity test, we observe that neural metrics inflate scores for text in scripts outside the encoder’s coverage and compress it into a narrow discriminative band. Switching to Meitei Mayek raises BERTScore and COMET by 21.6 and 23.3 points, respectively, though language and content are unchanged, and no metric preserves system ranking; on degenerate output the two retain 0.81 and 0.61 points on English-to-Indic directions. The effect reverses in the opposite direction. Metric choice should follow script coverage, not language identity. Lacking human judgments, our claims concern instrument behaviour, not human preference.

We describe team PANINI’s submission to the WMT 2026 Shared Task on Low-Resource Indic Language Translation. We submit primary systems for English↔Manipuri (Meiteilon) in both the Bengali (mni Beng) and Meitei Mayek (mni Mtei) scripts, covering all four directions. Rather than one shared multilingual adapter, we train four independent rank-stabilized LoRA (rsLoRA) adapters over IndicTrans2-1B, one per direction. Each adapter is trained on the authentic bitext plus three filtered synthetic streams the model generates for itself: back-translation, forward translation, and iterative pseudo-labelling, with subword regularisation and a script-aware processing pipeline. Our system obtains a COMET score of 92.31 and a BERTScore of 98.85 on English↔Manipuri (Meitei Mayek), the best in the direction. On the into-English directions, it reaches a COMET of 82.90 for mni Beng↔en and 80.20 for mniMtei↔en. These results show that per-direction adapters combined with self-generated synthetic data are effective even when authentic bitext is limited to a few thousand sentence pairs, and that a script-aware processing pipeline is necessary to produce well-formed output in both Manipuri scripts. The code and resources are released on GitHub.

Video-guided machine translation (VMT) translates text by leveraging the context provided by the accompanying video. This additional contextual information helps improve translation quality by disambiguating ambiguous words or explicitly indicating attributes such as the colour or gender of objects and people. Prior works have extensively explored VMT; however, most approaches train models from scratch, which requires large amounts of data. Additionally, the majority of VMT research has focused on the English–Chinese language pair. In this paper, we explore training strategies for VMT using only a limited amount of data and extend the task to Indic languages, specifically Hindi, Marathi and Odia. We also propose a synthetic data framework that is effective for training VMT models in low-resource language settings. Our results demonstrate an improvement of +1.75 BLEU compared to baseline models.

Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.

In recent years, machine translation has evolved with the integration of multimodal information. Infusion of multi-modality into translation tasks decreases ambiguation and enhances translation scores. Common modalities include images, speech, and videos, which provide additional context alongside the text to be translated. While multimodal translation with images has been extensively studied, video-guided machine translation (VMT) has gained increasing attention, particularly since Wang et al. 2019 first explored this task. In this paper, we provide a comprehensive overview of VMT, highlighting its unique challenges, methodologies, and recent advancements. Unlike previous surveys that primarily focus on image-guided multimodal machine translation, this work explores the distinct complexities and opportunities introduced by adding video as a modality to the translation task.