Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving external textual knowledge to answer visual queries. However, current methods often suffer from two critical misalignments during retrieval: distributional misalignment, driven by the severe stylistic shift from descriptive pre-training image captions to formal knowledge texts, and granularity misalignment, where lengthy knowledge documents contain excessive noise that drowns out key information, making it difficult to accurately match the visual query. To resolve these challenges, we propose RGCC, a Retriever-Guided Context Compression framework. By mining alignment-friendly data and further training a context compression model, RGCC condenses lengthy documents into compact, retriever-preferred representations that significantly improve cross-modal retrieval. Extensive experiments on the InfoSeek and E-VQA benchmarks demonstrate that RGCC achieves state-of-the-art retrieval performance and boosts downstream question-answering accuracy without additional online inference overhead.

Traditional research on Continual Machine Translation (CMT) has long been predicated on the assumption that catastrophic forgetting stems from the irreversible erasure of underlying linguistic knowledge. In this work, we challenge this conventional wisdom within the context of Large Language Models (LLMs). Through systematic behavioral and representational analyses, we demonstrate that the model’s latent knowledge often remains intact, yet its ability to align this knowledge with instructions becomes compromised. We attribute this " expressive drift" to the ubiquitous Supervised Fine-Tuning (SFT) paradigm. By minimizing token-level loss against fixed references, SFT neglects the inherent plurality of valid translations for a single source, inadvertently coercing the model toward narrow external distributions that conflict with its internal linguistic priors. To rectify this, we advocate for a transition from SFT to a Reinforcement Learning (RL) paradigm, which optimizes performance directly within the model’s own generative distribution. Our results show that RL significantly mitigates forgetting in both translation-specific and general tasks by preserving the model’s expressive stability. This work redefines the fundamental challenge of CMT and positions RL as a more principled approach for sustainable multilingual alignment in the era of LLMs.

Decoding language from the human brain remains a grand challenge for Brain-Computer Interfaces (BCIs). Current approaches typically rely on unimodal brain representations, neglecting the brain’s inherently multimodal processing. Inspired by the brain’s associative mechanisms, where viewing an image can evoke related sounds and linguistic representations, we propose a unified framework that leverages Multimodal Large Language Models (MLLMs) to align brain signals with a shared semantic space encompassing text, images, and audio. A router module dynamically selects and fuses modality-specific brain features according to the characteristics of each stimulus. Experiments on various fMRI datasets with textual, visual, and auditory stimuli demonstrate state-of-the-art performance, achieving an 8.48% average improvement on the most commonly used benchmark. We further extend our framework to EEG and MEG data, demonstrating flexibility and robustness across varying temporal and spatial resolutions. To our knowledge, this is the first unified BCI architecture capable of robustly decoding multimodal brain activity across diverse brain signals and stimulus types, offering a flexible solution for real-world applications.

Large Language Model (LLM)-based search agents have demonstrated superior performance in knowledge-intensive tasks through interleaved reasoning and search interactions. Process Reward Models (PRMs) can enhance search agent performance by guiding step selection during inference. However, existing PRMs for search agents face a critical bottleneck: assessment unreliability in their training data. This issue stems from an over-reliance on single-source supervision, forcing a compromise between global solvability and local validity. We propose CoPRM (Collaborative Process Reward Model), a process reward model trained on high-quality preference data generated through our Collaborative Calibration methodology. This approach synthesizes immediate quality assessments with outcome-informed evaluations through a dual-strategy mechanism: strictly enforcing consensus filtering to eliminate unreliable signals, and employing an outcome-informed gating mechanism to resolve heterogeneous cross-type trade-offs. Trained on 10k preference pairs, CoPRM provides universal test-time guidance, enabling diverse search agents to achieve consistent performance improvements across four multi-hop QA benchmarks, demonstrating substantial gains, robust cross-domain generalization, and superior efficiency.

Recent MLLMs have shown promise in scene-text-aware video understanding, yet their ability to understand scene text from the wearer’s perspective in AI-glasses videos remains underexplored. AI-glasses videos are passively captured without a camera-frame preview, limiting the wearer’s ability to adjust framing during recording. As a result, relevant text may be unstable, and the recorded view may not fully reflect the wearer’s attention. The challenge is thus not merely to read visible text, but to identify and read the right text given the wearer’s position, viewpoint trajectory, and goal. To systematically evaluate this capability, we introduce **WearerText**, a benchmark for **Wearer**-Centered Scene **Text** Understanding, comprising 1,101 real-world AI-glasses videos and 15,336 QA pairs spanning 13 tasks. These tasks are organized into a three-level diagnostic hierarchy: *L1 Text Perception*, *L2 Contextual Understanding*, and *L3 Wearer-Grounded Reasoning*, targeting unstable text perception, wearer-view alignment, and goal-oriented text reasoning, respectively. To ensure reliable construction and evaluation, we develop **MAH-V**, a multi-agent hybrid verification pipeline for QA construction, and propose **VerEval**, a Verified-evidence metric for open-ended Evaluation. Benchmarking 20 recent MLLMs shows that current models still struggle with wearer-centered scene text understanding, particularly on tasks requiring wearer-view alignment and goal-oriented text reasoning.

Video-guided Machine Translation (VMT) seeks to enhance translation quality by incorporating contextual information derived from paired short video clips. However, many VMT samples are text-sufficient; even when visual information is needed, only minimal cues are required. Aiming to tackle these issues, we propose a novel framework DART (Disambiguation-Aware Reasoning for Video-guided Machine Translation). Reinforcement learning is used to incorporate multimodal large language models’ multimodal reasoning into VMT. The model dynamically switches between text-only processing and multimodal integration, contingent on the necessity of visual disambiguation. Furthermore, we present TVRF (Translation-oriented Video Relevance Filtering), a systematic pipeline for constructing training data based on multimodal relevance to translation. This pipeline filters samples where video information is translation-relevant, mitigating training collapse caused by video-irrelevant data in conventional VMT. Experimental results show that our approach improves multimodal information utilization in VMT, yielding gains in both translation quality and computational efficiency.

Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks, and fine-tuning them for Machine Translation (MT) has improved their performance. However, vanilla fine-tuning often leads to catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks. These abilities, which are developed using proprietary and unavailable training data, make simple data replay methods ineffective. To overcome this issue, we propose a novel approach called Rationale Distillation. RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. These rationales connect prior knowledge with new tasks, acting as self-distillation targets to regulate the training process. By jointly training on reference translations and self-generated rationales, the model can learn new translation skills while preserving its general abilities across other tasks. Additionally, RaDis provides a fresh perspective on using rationales in the CL field and has the potential to serve as a general continual learning method for a variety of tasks.

Document Image Translation (DIT), which aims at translating documents in images from source language to the target, plays an important role in Document Intelligence. It requires a comprehensive understanding of document multi-modalities and a focused concentration on relevant textual regions during translation. However, most existing methods usually rely on the vanilla encoder-decoder paradigm, severely losing concentration on key regions that are especially crucial for complex-layout document translation. To tackle this issue, in this paper, we propose a new Query-Response DIT framework (QRDIT). QRDIT reformulates the DIT task into a parallel response/translation process of the multiple queries (i.e., relevant source texts), explicitly centralizing its focus toward the most relevant textual regions to ensure translation accuracy. A novel dynamic aggregation mechanism is also designed to enhance the text semantics in query features toward translation. Extensive experiments in four translation directions on three benchmarks demonstrate its state-of-the-art performance, showing significant translation quality improvements toward whole-page complex-layout document images.

Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model’s existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept “Bilingual Cognitive Advantage”. Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks. The code will be released upon acceptance.

Video-guided Machine Translation (VMT) aims to improve translation quality by integrating contextual information from paired short video clips. Mainstream VMT approaches typically incorporate multimodal information by uniformly sampling frames from the input videos. However, this paradigm frequently incurs significant computational overhead and introduces redundant multimodal content, which degrades both efficiency and translation quality. To tackle these challenges, we propose SHIFT (Selected Helpful Informative Frame for Translation). It is a lightweight, plug-and-play framework designed for VMT with Multimodal Large Language Models (MLLMs). SHIFT adaptively selects a single informative key frame when visual context is necessary; otherwise, it relies solely on textual input. This process is guided by a dedicated clustering module and a selector module. Experimental results demonstrate that SHIFT enhances the performance of MLLMs on the VMT task while simultaneously reducing computational cost, without sacrificing generalization ability. The code will be released upon acceptance.

Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix Modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an imageonly encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios. The code will be released upon acceptance.