Haiyang Yu

Author directory

Other people with similar names: Haiyang Yu

Unverified author pages with similar names: Haiyang Yu


2026

Deciphering scanned Oracle Bone Script (OBS) images into corresponding modern Chinese characters is a crucial step in AI-assisted OBS decipherment. As a highly pictographic writing system, OBS often embeds semantic information in glyph forms, making its decipherment depend on more than visual similarity alone. However, how to effectively exploit such glyph-meaning cues and how to make the decipherment process interpretable remain challenging. To this end, we propose an LVLM-based framework that bridges glyphs and meanings through structured glyph analysis. Specifically, we introduce a progressive training strategy that guides the model from radical analysis to pictographic analysis and finally to mutual analysis, enabling it to capture semantic information embedded in OBS glyphs. The generated textual analyses serve as both interpretable reasoning traces and explicit evidence for retrieving corresponding modern Chinese characters from a dictionary. We also construct the Pictographic Decipherment OBS Dataset (PD-OBS), which provides detailed radical and pictographic analyses to support glyph-analysis learning and dictionary-based retrieval. Experiments on OBS benchmarks demonstrate the strong zero-shot capability of our method, especially its substantially improved Top-10 accuracy on unseen OBS, highlighting its ability to provide reliable and interpretable candidate references for AI-assisted archaeological and historical studies. The code and dataset will be released at https://github.com/PKXX1943/PD-OBS.
Chinese ancient documents, invaluable carriers of millennia of Chinese history and culture, hold rich knowledge across diverse fields but face challenges in digitization and understanding—traditional methods only scan images, while current Vision-Language Models (VLMs) struggle with their visual/linguistic complexity. Existing document benchmarks focus on English printed texts or simplified Chinese, leaving a gap for evaluating VLMs on ancient Chinese documents. To address this, we present AncientDoc, the first benchmark for Chinese ancient documents, designed to assess VLMs from OCR to knowledge reasoning. AncientDoc includes five tasks (page-level OCR, vernacular translation, reasoning-based QA, knowledge-based QA, linguistic variant QA) and covers 14 document types, over 100 books, and about 3,000 pages. Based on AncientDoc, we evaluate mainstream VLMs using multiple metrics, supplemented by a human-aligned large language model for scoring.
The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks share one fundamental limitation: the attack itself is *static*. Template-based attacks freeze the image–text layout, while iterative sample-level attacks make only the image–text *content* dynamic, keeping the attack-iteration strategy *fixed* and the attacker’s parameters *frozen*. At the *meta level* they remain static, and we argue this partial dynamism cannot expose the full safety risks of multimodal models.We propose **Meta-Adaptive Multimodal Jailbreaking (MAMJ)**, which lifts adversarial dynamism from the sample level to the meta level by making the attacker’s policy itself the optimisation target along two axes: an **attack strategy prompt (ASP)** 𝜃 in language space, governing how the attack is iterated against a victim VLM, and the **attacker weights** 𝜙, determining the success rate attainable under a given 𝜃.Evolving over a *group of multimodal attack trajectories*, an LLM-based critique of failed trajectories first refines 𝜃, after which group-aggregated attack-success-rate (ASR) rewards update 𝜙. On MM-SafetyBench, MAMJ attains 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, exceeding the strongest sample-level baseline by up to 24.1 percentage points, and the learned attacker (𝜃⋆, 𝜙⋆) transfers *without retraining* to unseen victims while remaining effective under representative defences.These results expose a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and underscore the need for defences that anticipate meta-level adversaries. Code and checkpoints will be released.
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.

2025

Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout andText in a Large Language Model (LayTextLLM) for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at URL masked for anonymous review.