Huahui Yi
Author directory2026
ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoning
Gongli Xi | Kun Wang | Zeming Gao | Huahui Yi | Haolang Lu | Qianshan Wei | Ye Tian | Wendong Wang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Gongli Xi | Kun Wang | Zeming Gao | Huahui Yi | Haolang Lu | Qianshan Wei | Ye Tian | Wendong Wang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large multimodal reasoning models solve challenging visual problems via explicit long-chain inference: they gather visual clues from images and decode clues into textual tokens. Yet this capability also increases hallucinations, where the model generates content that is not supported by the input image or the question. To understand this failure mode, we identify reasoning drift: during clue gathering, the model over-focuses on question-irrelevant entities, diluting focus on task-relevant cues and gradually decoupling the reasoning trace from visual grounding. As a consequence, many inference-time localization or intervention methods developed for non-reasoning models fail to pinpoint the true clues in reasoning settings. Motivated by these insights, we introduce ClueRecall, a metric for assessing visual clue retrieval, and present ClueTracer, a training-free, parameter-free, and architecture-agnostic plugin for hallucination suppression. ClueTracer starts from the question and traces how key clues propagate along the model’s reasoning pathway (question → outputs → visual tokens), thereby localizing task-relevant patches while suppressing spurious attention to irrelevant regions. Remarkably, without any additional training, ClueTracer improves all reasoning architectures (including R1-OneVision, Ocean-R1, MM-Eureka, .) by 1.21 on reasoning benchmarks.
SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics
Haolong Hu | Hanyu Li | Tiancheng He | Huahui Yi | An Zhang | Qiankun Li | Kun Wang | Yang Liu | Zhigang Zeng
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Haolong Hu | Hanyu Li | Tiancheng He | Huahui Yi | An Zhang | Qiankun Li | Kun Wang | Yang Liu | Zhigang Zeng
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
MLLMs are increasingly deployed in multi-turn settings, where attackers can escalate unsafe intent through the evolving visual-text history and exploit long-context safety decay. Yet safety alignment is still dominated by single-turn data and fixed-template dialogues, leaving a mismatch between training and deployment.
To bridge this gap, we propose SaFeR-Steer, a progressive multi-turn alignment framework that combines staged synthetic bootstrapping with tutor-in-the-loop GRPO to train a single student under adaptive, on-policy attacks. We also introduce Trajectory-Consistent Summative Reward (TCSR), which aggregates the historical minimum and average of turn rewards so that any low-quality turn affects the trajectory-level return.
I. Dataset. We release STEER, a multi-turn multimodal safety dataset with STEER-SFT (12,934), STEER-RL (2,000), and STEER-Bench (3,227) dialogues spanning 1–10 turns. II. Experiment. Starting from Qwen2.5-VL-3B/7B, SaFeR-Steer substantially improves Safety/Helpfulness on both single-turn (48.30/45.86 → 81.84/70.77 for 3B; 56.21/60.32 → 87.89/77.40 for 7B) and multi-turn benchmarks (12.55/27.13 → 55.58/70.27 for 3B; 24.66/46.48 → 64.89/72.35 for 7B), shifting failures to later turns and yielding robustness beyond scaling alone.
Code is available at https://github.com/Ed-Bg/SaFeR-Steer-full
To bridge this gap, we propose SaFeR-Steer, a progressive multi-turn alignment framework that combines staged synthetic bootstrapping with tutor-in-the-loop GRPO to train a single student under adaptive, on-policy attacks. We also introduce Trajectory-Consistent Summative Reward (TCSR), which aggregates the historical minimum and average of turn rewards so that any low-quality turn affects the trajectory-level return.
I. Dataset. We release STEER, a multi-turn multimodal safety dataset with STEER-SFT (12,934), STEER-RL (2,000), and STEER-Bench (3,227) dialogues spanning 1–10 turns. II. Experiment. Starting from Qwen2.5-VL-3B/7B, SaFeR-Steer substantially improves Safety/Helpfulness on both single-turn (48.30/45.86 → 81.84/70.77 for 3B; 56.21/60.32 → 87.89/77.40 for 7B) and multi-turn benchmarks (12.55/27.13 → 55.58/70.27 for 3B; 24.66/46.48 → 64.89/72.35 for 7B), shifting failures to later turns and yielding robustness beyond scaling alone.
Code is available at https://github.com/Ed-Bg/SaFeR-Steer-full
SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safety
Zixuan Xu | Tiancheng He | Huahui Yi | Kun Wang | Xi Chen | Gongli Xi | Qiankun Li | Kang Li | Yang Liu | Zhigang Zeng
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Zixuan Xu | Tiancheng He | Huahui Yi | Kun Wang | Xi Chen | Gongli Xi | Qiankun Li | Kang Li | Yang Liu | Zhigang Zeng
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Vision–language models remain susceptible to multimodal jailbreaks and over-refusal because safety hinges on both visual evidence and user intent, while many alignment pipelines supervise only the final response. To address this, we present SaFeR-ToolKit, which formalizes safety decision-making as an inspectable protocol: during offline data synthesis, a planner specifies a persona, a **Perception**→**Reasoning**→**Decision** tool set, and a constrained transition graph, while a responder emits a typed key–value tool trace before the final answer. To learn this protocol, we train a single policy with a three-stage curriculum (SFT→DPO→GRPO), where GRPO directly supervises tool usage beyond answer-level feedback. We contribute (**I**) the first tool-based safety reasoning dataset of 30,654 training entries (SFT 6k, DPO 18.6k, GRPO 6k) plus 1k held-out evaluation, and (**II**) experiments on Qwen2.5-VL showing large gains in Safety/Helpfulness/Reasoning Rigor on 3B (29.39/45.04/4.98 → 84.40/71.13/78.87) and 7B (53.21/52.92/19.26 → 86.34/80.79/85.34), while preserving general capabilities (3B: 58.67 → 59.21; 7B: 66.39 → 66.81). Code: [https://github.com/Duebassx/SafeR_Toolkit](https://github.com/Duebassx/SafeR_Toolkit).
2025
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations
Kangyu Zhu | Ziyuan Qin | Huahui Yi | Zekun Jiang | Qicheng Lao | Shaoting Zhang | Kang Li
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Kangyu Zhu | Ziyuan Qin | Huahui Yi | Zekun Jiang | Qicheng Lao | Shaoting Zhang | Kang Li
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
While mainstream vision-language models (VLMs) have advanced rapidly in understanding image-level information, they still lack the ability to focus on specific areas designated by humans. Rather, they typically rely on large volumes of high-quality image-text paired data to learn and generate posterior attention maps. To address this critical issue, we propose leveraging visual prompts—simple visual markers in various forms—to guide and enhance the formation of region-specific attention. Thus, we introduce MedVP, a pioneering framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation for visual prompt-guided fine-tuning. We successfully outperform recent state-of-the-art large models across multiple medical VQA datasets. Extensive experiments and Human evaluation are conducted to analyze the impact of different visual prompt forms and how they contribute to performance improvement. The results demonstrate both the effectiveness and clinical significance of our approach.