Shiguang Shan
Author directory2026
UniSurgBench: Benchmarking Surgical Video Understanding with Step-wise Reasoning and Workflow Aligned Evaluation
Zhenxin Lei | Mengyang Li | Qian Li | Haozhe Cui | Zhimin Zhao | Hu Han | Shiguang Shan | Xilin Chen
Findings of the Association for Computational Linguistics: EMNLP 2026
Zhenxin Lei | Mengyang Li | Qian Li | Haozhe Cui | Zhimin Zhao | Hu Han | Shiguang Shan | Xilin Chen
Findings of the Association for Computational Linguistics: EMNLP 2026
Multimodal large language models have made substantial progress in medical visual understanding, but temporal reasoning and knowledge-guided reasoning over real surgical videos remain underexplored. Existing surgical benchmarks mainly focus on static frames, single procedures, or simple questions, which limits their ability to capture the complexity of real clinical workflows. To address this gap, we introduce UniSurgBench, a unified benchmark for open-ended surgical video reasoning, comprising 2,302 question-answer samples collected from real surgical videos spanning 45 procedures and 29 anatomical structures. Each sample is annotated with temporally grounded evidence and structured reasoning steps. We also propose SurgExpEval, a two-stage evaluation protocol that extracts structured reasoning components from free-form model outputs and jointly assesses answer accuracy, reasoning quality, and temporal grounding. Evaluations with 23 proprietary, open-source, and medical-related MLLMs show that MLLMs can produce clinically reasonable and causally coherent responses. However, they often fail to associate these responses with fine-grained visual evidence, highlighting evidence-level visual grounding as a key bottleneck for reliable surgical intelligence.
Probe to Act: Elevating Browser-Use Agent via Active Visual Probing
Keliang Li | Heng Wang | Chen Hu | Daxin Jiang | Hong Chang | Shiguang Shan
Findings of the Association for Computational Linguistics: EMNLP 2026
Keliang Li | Heng Wang | Chen Hu | Daxin Jiang | Hong Chang | Shiguang Shan
Findings of the Association for Computational Linguistics: EMNLP 2026
Browser-use agents require seamless alignment between structured web metadata and visual information, while preserving relevant context across long interactions. Existing interfaces often rely on either screenshot-level action prediction or static Set-of-Marks overlays, leaving the model to resolve dense DOM-pixel alignment before every operation. We introduce Probe to Act (P2A), an active probing framework for the browser-agent loop that moves this alignment into decision time. P2A addresses an asymmetric bridge between symbolic DOM hypotheses and screenshot layout by rendering on-demand symbolic DOM structure back into pixels. Before committing a state-changing browser operation, the agent can issue lightweight probes to translate DOM handles into pixel evidence, map screen regions back to DOM candidates, register visual-only targets, and commit verified notes. These interleaved processes naturally produce evidence-based memory: only probed, acted-on, or explicitly committed observations are kept across steps, preserving only decision-critical evidence in long-horizon contexts. P2A can be used as a prompting strategy for proprietary models under the standard DOM+SoM interface, and can be distilled into open-weight models through cold-start synthesis and self-bootstrapped SFT. Across three browser-use benchmarks, P2A shows clear gains on task success rate for both proprietary and fine-tuned models; on VisualWebArena, for example, it improves Gemini-3-Pro from 54.1% to 61.2% and Qwen3-VL-8B from 24.6% to 32.9%, while matching the costly full-observation history (3̃×) at near context cost of action-only history (1̃.2×).
FlexLTS: A Unified Paradigm for Content-Accurate Lip-to-Speech Synthesis with Optional Lip Reading
Zhechen Liu | Shuang Yang | Shiguang Shan | Xilin Chen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zhechen Liu | Shuang Yang | Shiguang Shan | Xilin Chen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Lip-to-speech (LTS) synthesis aims to generate intelligible speech from silent lip movements, but the ambiguous visual-to-phoneme mapping makes content-accurate synthesis challenging. Existing methods mainly follow two routes: (1) direct synthesis from lip movements and (2) synthesis dependent on an external lip-reading model. The former often suffers from content errors, while the latter improves content accuracy at the cost of computational overhead and error propagation. To address these limitations, we introduce FlexLTS, a unified paradigm for content-accurate LTS with optional lip reading. It generates accurate speech without requiring lip reading, while still benefiting from lip-reading text when available. FlexLTS learns phonetic prior embeddings from training transcripts and employs a visual-centric prior fusion mechanism to selectively incorporate relevant phonetic priors into the visual stream, yielding content-enriched semantic representations. Moreover, our proposed Progressive Textual Dropout strategy further promotes prior internalization by reducing reliance on explicit text during training. Experiments show that FlexLTS achieves state-of-the-art content accuracy and speech quality on the popular LRS2-BBC and LRS3-TED benchmarks and also performs well in cross-dataset zero-shot evaluation. Notably, without using any lip-reading model, FlexLTS surpasses existing lip-reading-aided methods in content accuracy.
INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs
Yangjunqi | Yuecong Min | Jie Zhang | Shiguang Shan | Xilin Chen
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yangjunqi | Yuecong Min | Jie Zhang | Shiguang Shan | Xilin Chen
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Despite rapid progress, Video Large Language Models (Video-LLMs) remain unreliable due to hallucinations, which are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality).Existing benchmarks provide limited coverage of factuality hallucinations and predominantly evaluate models only in clean settings.We introduce INFACT, a diagnostic benchmark comprising 9,800 QA instances with fine-grained taxonomies for faithfulness and factuality, spanning real and synthetic videos.INFACT evaluates models in four modes: Base (clean), Visual Degradation, Evidence Corruption, and Temporal Intervention for order-sensitive items.Reliability under induced modes is quantified using Resist Rate (RR) and Temporal Sensitivity Score (TSS).Experiments on 14 representative Video-LLMs reveal that higher Base-mode accuracy does not reliably translate to higher reliability in the induced modes, with evidence corruption reducing stability and temporal intervention yielding the largest degradation.Notably, many open-source baselines exhibit near-zero TSS on factuality, indicating pronounced temporal inertia on order-sensitive questions.
2025
Evaluating Cognitive-Behavioral Fixation via Multimodal User Viewing Patterns on Social Media
Yujie Wang | Yunwei Zhao | Jing Yang | Han Han | Shiguang Shan | Jie Zhang
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Yujie Wang | Yunwei Zhao | Jing Yang | Han Han | Shiguang Shan | Jie Zhang
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Digital social media platforms frequently contribute to cognitive-behavioral fixation, a phenomenon in which users exhibit sustained and repetitive engagement with narrow content domains. While cognitive-behavioral fixation has been extensively studied in psychology, methods for computationally detecting and evaluating such fixation remain underexplored. To address this gap, we propose a novel framework for assessing cognitive-behavioral fixation by analyzing users’ multimodal social media engagement patterns. Specifically, we introduce a multimodal topic extraction module and a cognitive-behavioral fixation quantification module that collaboratively enable adaptive, hierarchical, and interpretable assessment of user behavior. Experiments on existing benchmarks and a newly curated multimodal dataset demonstrate the effectiveness of our approach, laying the groundwork for scalable computational analysis of cognitive fixation. All code in this project is publicly available for research purposes at https://github.com/Liskie/cognitive-fixation-evaluation.