Recent OCR models achieve strong results on document benchmarks, with several 1–2B OCR-specialized models surpassing much larger general multimodal LLMs (MLLMs). However, complex real-world documents often contain blur, cropping, curved text, handwriting, and unfamiliar structures, which remain challenging for current OCR systems. We trace this gap to a shift in the error source: errors on canonical benchmarks mainly arise from layout analysis, including localization, element classification, and reading order, while errors under these challenging conditions mainly arise from content recognition. To this end, we propose CL-OCR, a two-stage framework that first fuses semantic structure from an MLLM-based detector with geometric candidates from a vision detector to obtain calibrated layouts, then enhances recognition via a shared LoRA adapter gated by a lightweight router. We further construct OCRWildBench and a companion training split to study these conditions. Experiments show that CL-OCR remains competitive on canonical benchmarks and achieves the strongest overall performance among OCR-specialized models on OCRWildBench.

The computational and memory overheads associated with expanding the context window of LLMs severely limit their scalability. A noteworthy solution is vision-text compression (VTC), exemplified by frameworks like DeepSeek-OCR and Glyph, which convert long texts into dense 2D visual representations, thereby achieving token compression ratios of 2x–10x. However, the impact of this high information density on the core long-context capabilities of vision-language models (VLMs) remains under-investigated. To address this gap, we introduce the first benchmark for VTC and systematically assess the performance of VLMs across three long-context understanding settings: VTC-Retrieval, VTC-Reasoning, and VTC-Memory. We further introduce VTCBench-Wild to evaluate robustness across diverse rendering configurations. We comprehensively evaluate leading open-weight and proprietary models on our benchmarks. The results indicate that several VLMs can retrieve information from rendered text in short-to-moderate contexts, but this ability degrades at longer lengths, and the degradation is especially pronounced on tasks requiring binding and integrating dispersed evidence. This study provides diagnostic evidence of VTC and serves as a foundation for designing more efficient and scalable VLMs.

Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed instruction datasets. However, novel tasks would be encountered sequentially in dynamic world, which urges for equipping LMMs with multimodal continual instruction learning (MCIT) ability especially for diverse and challenging generative tasks. Existing MCIT methods do not fully exploit the unique attribute of LMMs and often gain performance at the expense of efficiency. In this paper, we propose a novel prompt learning framework for MCIT to effectively alleviate forgetting of previous knowledge while managing computational complexity with natural image-text supervision. Concretely, we learn prompts for each task and exploit efficient prompt fusion for knowledge transfer and prompt selection for complexity management with dual-modality guidance. Extensive experiments demonstrate that our approach achieves substantial +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation.

Instruction tuning is widely used to enhance a pre-trained Multimodal Large Language Model (MLLM) to understand and follow human instructions by training it on a curated set of task-specific dataset. However, it is infeasible to collect all possible instruction datasets simultaneously in real-world scenarios. Thus, enabling MLLM with continual instruction tuning is essential for maintaining their adaptability. However, existing methods often trade off memory efficiency for performance gains, significantly compromising overall efficiency. In this paper, we propose a task-specific expansion and task-general fusion framework based on the variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets. Furthermore, we analyze the information leakage present in the existing benchmark and propose a new and more challenging benchmark to rationally evaluate the performance of different methods. Comprehensive experiments showcase a significant performance improvement of our method compared to existing state-of-the-art methods. Our code will be public available.