Yao Zhang
Author directoryOther people with similar names: Yao Zhang
Unverified author pages with similar names: Yao Zhang
2026
Regret-Now: A Physics-Inspired Regret Framework for Temporal Knowledge Graph Question Answering with LLMs
Danyu Huang | Yao Zhang | Jun Wang | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2026
Danyu Huang | Yao Zhang | Jun Wang | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2026
Large Language Models have achieved impressive results in general reasoning tasks. However, they still face significant challenges when applied to temporal knowledge graph question answering (TKGQA), particularly exhibiting broken temporal reasoning chains and a lack of dynamic error-correction. These limitations hinder their capacity to handle complex temporal logic and make it difficult to recover once a reasoning error occurs. To address this issue, we propose Regret-Now, a novel LLM-based temporal reasoning framework inspired by the physical principle of minimum potential energy. Regret-Now models the reasoning process as a dynamic trajectory moving toward a more stable state, where each step is expected to a lower potential energy. We introduce the Regret Stage that evaluates the “potential energy” of each intermediate reasoning step and triggers real-time rollback if an abnormal rise in potential energy is detected—indicating a likely error. We evaluate Regret-Now on two standard TKGQA benchmarks: CronQuestions and MultiTQ. Experimental results show consistent gains over strong baselines, validating physics-inspired modeling for LLM-based TKGQA. The code can be found at https://github.com/h-yii/Regret-Now.
M-TRACE: Detecting and Mitigating Time-Anchor Drift via Step-wise Conflict Checking in Temporal Reasoning
Danyu Huang | Jiayuan Jiang | Yao Zhang | Jun Wang | Huijia Li | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2026
Danyu Huang | Jiayuan Jiang | Yao Zhang | Jun Wang | Huijia Li | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2026
As the real world continuously evolves, temporal facts change over time, requiring large language models to simultaneously rely on internal parametric knowledge and externally retrieved evidence for temporal reasoning. However, external knowledge may be inaccurate, while internal knowledge can become outdated. Temporal inconsistencies between these heterogeneous sources can accumulate during multi-step reasoning, leading to Time-Anchor Drift (TAD)—a phenomenon where an incorrect temporal reference is established early and subsequently propagated, ultimately causing reasoning failure. To address this issue, we propose M-TRACE, a multi-agent reasoning framework for temporal knowledge conflicts. M-TRACE explicitly maintains a State Timeline to perform step-wise temporal alignment and coexistence checks between internal states and external evidence. Detected conflicts are summarized into a structured Conflict Report, which guides conflict-aware final reasoning. We further introduce TimeConfQA, a temporal question answering benchmark with controlled temporal knowledge conflicts. Experimental results show that M-TRACE effectively reduces time-anchor drift and consistently improves performance on complex temporal question answering tasks, demonstrating the value of explicit conflict modeling for temporal reasoning. The code can be found at https://github.com/h-yii/M-TRACE.
MSIA: Adaptive Medical Multimodal Multi-turn Semantic Jailbreak
Zhiheng Han | Yao Zhang | Jun Wang | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2026
Zhiheng Han | Yao Zhang | Jun Wang | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2026
Medical multimodal large language models are increasingly deployed in high-stakes clinical settings, yet current safety evaluations largely overlook a critical failure mode: covert semantic drift that accumulates across clinically plausible multi-turn interactions. Such drift can lead models to gradually exaggerate or conceal critical medical findings without triggering explicit safety mechanisms. We propose MSIA (Medical Semantic Infiltration Attack), a framework for modeling and inducing multi-turn medical semantic jailbreaks in clinical dialogues. MSIA enables the controlled optimization of cumulative semantic drift under stealth constraints through adaptive strategy selection and closed-loop reward feedback grounded in medical evidence. Experiments on chest X-ray–based multimodal medical dialogues show that MSIA consistently outperforms existing jailbreak methods across GPT-4o, Claude, and Gemini, achieving an average attack success rate of 76.67%. These results expose substantial and previously underestimated vulnerabilities of medical LLMs in realistic multi-turn clinical interactions. Code is available here: https://github.com/HeYamo/MSIA.
2025
Generating Questions, Answers, and Distractors for Videos: Exploring Semantic Uncertainty of Object Motions
Wenjian Ding | Yao Zhang | Jun Wang | Adam Jatowt | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2025
Wenjian Ding | Yao Zhang | Jun Wang | Adam Jatowt | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2025
Video Question-Answer-Distractors (QADs) show promising values for assessing the performance of systems in perceiving and comprehending multimedia content. Given the significant cost and labor demands of manual annotation, existing large-scale Video QADs benchmarks are typically generated automatically using video captions. Since video captions are incomplete representations of visual content and susceptible to error propagation, direct generation of QADs from video is crucial. This work first leverages a large vision-language model for video QADs generation. To enhance the consistency and diversity of the generated QADs, we propose utilizing temporal motion to describe the video objects. In addition, We design a selection mechanism that chooses diverse temporal object motions to generate diverse QADs focusing on different objects and interactions, maximizing overall semantic uncertainty for a given video. Evaluation on the NExT-QA and Perception Test benchmarks demonstrates that the proposed approach significantly improves both the consistency and diversity of QADs generated by a range of large vision-language models, thus highlighting its effectiveness and generalizability.
Listening to Patients: Detecting and Mitigating Patient Misreport in Medical Dialogue System
Lang Qin | Yao Zhang | Hongru Liang | Adam Jatowt | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2025
Lang Qin | Yao Zhang | Hongru Liang | Adam Jatowt | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2025
Medical Dialogue Systems (MDSs) have emerged as promising tools for automated healthcare support through patient-agent interactions. Previous efforts typically relied on an idealized assumption — patients can accurately report symptoms aligned with their actual health conditions. However, in reality, patients often misreport their symptoms, due to cognitive limitations, emotional factors, etc. Overlooking patient misreports can significantly compromise the diagnostic accuracy of MDSs. To address this critical issue, we emphasize the importance of enabling MDSs to “listen to patients” by tackling two key challenges: how to detect misreport and mitigate misreport effectively. In this work, we propose PaMis, a novel framework that can detect patient misreports based on calculating the structural entropy of the dialogue entity graph, and mitigate them through generating controlled clarifying questions. Our experimental results demonstrate that PaMis effectively enhances MDSs reliability by effectively addressing patient misreports during the medical response generation process.
SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View
Yongjie Xiao | Hongru Liang | Peixin Qin | Yao Zhang | Wenqiang Lei
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yongjie Xiao | Hongru Liang | Peixin Qin | Yao Zhang | Wenqiang Lei
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Despite the great potential of large language models (LLMs) in machine comprehension, it is still disturbing to fully count on them in real-world scenarios. This is probably because there is no rational explanation for whether the comprehension process of LLMs is aligned with that of experts. In this paper, we propose SCOP to carefully examine how LLMs perform during the comprehension process from a cognitive view. Specifically, it is equipped with a systematical definition of five requisite skills during the comprehension process, a strict framework to construct testing data for these skills, and a detailed analysis of advanced open-sourced and closed-sourced LLMs using the testing data. With SCOP, we find that it is still challenging for LLMs to perform an expert-level comprehension process. Even so, we notice that LLMs share some similarities with experts, e.g., performing better at comprehending local information than global information. Further analysis reveals that LLMs can be somewhat unreliable — they might reach correct answers through flawed comprehension processes. Based on SCOP, we suggest that one direction for improving LLMs is to focus more on the comprehension process, ensuring all comprehension skills are thoroughly developed during training.
2024
Exploring Union and Intersection of Visual Regions for Generating Questions, Answers, and Distractors
Wenjian Ding | Yao Zhang | Jun Wang | Adam Jatowt | Zhenglu Yang
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Wenjian Ding | Yao Zhang | Jun Wang | Adam Jatowt | Zhenglu Yang
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Multiple-choice visual question answering (VQA) is to automatically choose a correct answer from a set of choices after reading an image. Existing efforts have been devoted to a separate generation of an image-related question, a correct answer, or challenge distractors. By contrast, we turn to a holistic generation and optimization of questions, answers, and distractors (QADs) in this study. This integrated generation strategy eliminates the need for human curation and guarantees information consistency. Furthermore, we first propose to put the spotlight on different image regions to diversify QADs. Accordingly, a novel framework ReBo is formulated in this paper. ReBo cyclically generates each QAD based on a recurrent multimodal encoder, and each generation is focusing on a different area of the image compared to those already concerned by the previously generated QADs. In addition to traditional VQA comparisons with state-of-the-art approaches, we also validate the capability of ReBo in generating augmented data to benefit VQA models.
2022
Fact-Tree Reasoning for N-ary Question Answering over Knowledge Graphs
Yao Zhang | Peiyao Li | Hongru Liang | Adam Jatowt | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2022
Yao Zhang | Peiyao Li | Hongru Liang | Adam Jatowt | Zhenglu Yang
Findings of the Association for Computational Linguistics: ACL 2022
Current Question Answering over Knowledge Graphs (KGQA) task mainly focuses on performing answer reasoning upon KGs with binary facts. However, it neglects the n-ary facts, which contain more than two entities. In this work, we highlight a more challenging but under-explored task: n-ary KGQA, i.e., answering n-ary facts questions upon n-ary KGs. Nevertheless, the multi-hop reasoning framework popular in binary KGQA task is not directly applicable on n-ary KGQA. We propose two feasible improvements: 1) upgrade the basic reasoning unit from entity or relation to fact, and 2) upgrade the reasoning structure from chain to tree. Therefore, we propose a novel fact-tree reasoning framework, FacTree, which integrates the above two upgrades. FacTree transforms the question into a fact tree and performs iterative fact reasoning on the fact tree to infer the correct answer. Experimental results on the n-ary KGQA dataset we constructed and two binary KGQA benchmarks demonstrate the effectiveness of FacTree compared with state-of-the-art methods.
Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension
Huibin Zhang | Zhengkun Zhang | Yao Zhang | Jun Wang | Yufan Li | Ning Jiang | Xin Wei | Zhenglu Yang
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Huibin Zhang | Zhengkun Zhang | Yao Zhang | Jun Wang | Yufan Li | Ning Jiang | Xin Wei | Zhenglu Yang
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Procedural Multimodal Documents (PMDs) organize textual instructions and corresponding images step by step. Comprehending PMDs and inducing their representations for the downstream reasoning tasks is designated as Procedural MultiModal Machine Comprehension (M3C). In this study, we approach Procedural M3C at a fine-grained level (compared with existing explorations at a document or sentence level), that is, entity. With delicate consideration, we model entity both in its temporal and cross-modal relation and propose a novel Temporal-Modal Entity Graph (TMEG). Specifically, graph structure is formulated to capture textual and visual entities and trace their temporal-modal evolution. In addition, a graph aggregation module is introduced to conduct graph encoding and reasoning. Comprehensive experiments across three Procedural M3C tasks are conducted on a traditional dataset RecipeQA and our new dataset CraftQA, which can better evaluate the generalization of TMEG.