Haiyang Yu
Author directoryOther people with similar names: Haiyang Yu
Unverified author pages with similar names: Haiyang Yu
2026
Interpretable Oracle Bone Script Decipherment via Progressive Glyph Analysis with LVLMs
Kaixin Peng | Mengyang Zhao | Haiyang Yu | Teng Fu | Bin Li
Findings of the Association for Computational Linguistics: EMNLP 2026
Kaixin Peng | Mengyang Zhao | Haiyang Yu | Teng Fu | Bin Li
Findings of the Association for Computational Linguistics: EMNLP 2026
Deciphering scanned Oracle Bone Script (OBS) images into corresponding modern Chinese characters is a crucial step in AI-assisted OBS decipherment. As a highly pictographic writing system, OBS often embeds semantic information in glyph forms, making its decipherment depend on more than visual similarity alone. However, how to effectively exploit such glyph-meaning cues and how to make the decipherment process interpretable remain challenging. To this end, we propose an LVLM-based framework that bridges glyphs and meanings through structured glyph analysis. Specifically, we introduce a progressive training strategy that guides the model from radical analysis to pictographic analysis and finally to mutual analysis, enabling it to capture semantic information embedded in OBS glyphs. The generated textual analyses serve as both interpretable reasoning traces and explicit evidence for retrieving corresponding modern Chinese characters from a dictionary. We also construct the Pictographic Decipherment OBS Dataset (PD-OBS), which provides detailed radical and pictographic analyses to support glyph-analysis learning and dictionary-based retrieval. Experiments on OBS benchmarks demonstrate the strong zero-shot capability of our method, especially its substantially improved Top-10 accuracy on unseen OBS, highlighting its ability to provide reliable and interpretable candidate references for AI-assisted archaeological and historical studies. The code and dataset will be released at https://github.com/PKXX1943/PD-OBS.
Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning
Haiyang Yu | Yuchuan Wu | Fan Shi | Jinghui Lu | Ke Niu | Xiaodong Ge | Minghan Zhuo | Jingqun Tang | Bin Li
Findings of the Association for Computational Linguistics: ACL 2026
Haiyang Yu | Yuchuan Wu | Fan Shi | Jinghui Lu | Ke Niu | Xiaodong Ge | Minghan Zhuo | Jingqun Tang | Bin Li
Findings of the Association for Computational Linguistics: ACL 2026
Chinese ancient documents, invaluable carriers of millennia of Chinese history and culture, hold rich knowledge across diverse fields but face challenges in digitization and understanding—traditional methods only scan images, while current Vision-Language Models (VLMs) struggle with their visual/linguistic complexity. Existing document benchmarks focus on English printed texts or simplified Chinese, leaving a gap for evaluating VLMs on ancient Chinese documents. To address this, we present AncientDoc, the first benchmark for Chinese ancient documents, designed to assess VLMs from OCR to knowledge reasoning. AncientDoc includes five tasks (page-level OCR, vernacular translation, reasoning-based QA, knowledge-based QA, linguistic variant QA) and covers 14 document types, over 100 books, and about 3,000 pages. Based on AncientDoc, we evaluate mainstream VLMs using multiple metrics, supplemented by a human-aligned large language model for scoring.
Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
Shen Pang | Benlei Cui | Yuke Wang | Xuemei Dong | Yuwen Zhai | Jingqun Tang | Haiyang Yu | Hui Xue | Longtao Huang | Haiwen Hong
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Shen Pang | Benlei Cui | Yuke Wang | Xuemei Dong | Yuwen Zhai | Jingqun Tang | Haiyang Yu | Hui Xue | Longtao Huang | Haiwen Hong
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks share one fundamental limitation: the attack itself is *static*. Template-based attacks freeze the image–text layout, while iterative sample-level attacks make only the image–text *content* dynamic, keeping the attack-iteration strategy *fixed* and the attacker’s parameters *frozen*. At the *meta level* they remain static, and we argue this partial dynamism cannot expose the full safety risks of multimodal models.We propose **Meta-Adaptive Multimodal Jailbreaking (MAMJ)**, which lifts adversarial dynamism from the sample level to the meta level by making the attacker’s policy itself the optimisation target along two axes: an **attack strategy prompt (ASP)** 𝜃 in language space, governing how the attack is iterated against a victim VLM, and the **attacker weights** 𝜙, determining the success rate attainable under a given 𝜃.Evolving over a *group of multimodal attack trajectories*, an LLM-based critique of failed trajectories first refines 𝜃, after which group-aggregated attack-success-rate (ASR) rewards update 𝜙. On MM-SafetyBench, MAMJ attains 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, exceeding the strongest sample-level baseline by up to 24.1 percentage points, and the learned attacker (𝜃⋆, 𝜙⋆) transfers *without retraining* to unseen victims while remaining effective under representative defences.These results expose a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and underscore the need for defences that anticipate meta-level adversaries. Code and checkpoints will be released.
MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
Yuchi Wang | Dingkang Yang | Haiyang Yu | Weikang Bian | Jiefeng Long | Liang Xiao | Chao Feng | Hongsheng Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yuchi Wang | Dingkang Yang | Haiyang Yu | Weikang Bian | Jiefeng Long | Liang Xiao | Chao Feng | Hongsheng Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.
2025
A Bounding Box is Worth One Token - Interleaving Layout and Text in a Large Language Model for Document Understanding
Jinghui Lu | Haiyang Yu | Yanjie Wang | Yongjie Ye | Jingqun Tang | Ziwei Yang | Binghong Wu | Qi Liu | Hao Feng | Han Wang | Hao Liu | Can Huang
Findings of the Association for Computational Linguistics: ACL 2025
Jinghui Lu | Haiyang Yu | Yanjie Wang | Yongjie Ye | Jingqun Tang | Ziwei Yang | Binghong Wu | Qi Liu | Hao Feng | Han Wang | Hao Liu | Can Huang
Findings of the Association for Computational Linguistics: ACL 2025
Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout andText in a Large Language Model (LayTextLLM) for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at URL masked for anonymous review.
Search
Fix author
Co-authors
- Jingqun Tang 3
- Bin Li 2
- Jinghui Lu 2
- Weikang Bian 1
- Benlei Cui 1
- Xuemei Dong 1
- Chao Feng 1
- Hao Feng 1
- Teng Fu 1
- Xiaodong Ge 1
- Haiwen Hong 1
- Can Huang 1
- Longtao Huang 1
- Hongsheng Li 1
- Hao Liu 1
- Qi Liu 1
- Jiefeng Long 1
- Ke Niu 1
- Shen Pang 1
- Kaixin Peng 1
- Fan Shi 1
- Han Wang 1
- Yanjie Wang 1
- Yuchi Wang 1
- Yuke Wang 1
- Binghong Wu 1
- Yuchuan Wu 1
- Liang Xiao 1
- Hui Xue 1
- Dingkang Yang 1
- Ziwei Yang 1
- Yongjie Ye 1
- Yuwen Zhai 1
- Mengyang Zhao 1
- Minghan Zhuo 1