Zhihui Li
Author directory2026
ToM-Video: A Theory-of-Mind Video Benchmark with Spatio-Temporal Grounded Reasoning
Quanling Liu | Zizhuo Lin | Jinsheng Quan | Xin Shi | Xueqiao Zhang | Chao Zhang | Yifan Zhu | Zhihui Li | Yawei Luo
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Quanling Liu | Zizhuo Lin | Jinsheng Quan | Xin Shi | Xueqiao Zhang | Chao Zhang | Yifan Zhu | Zhihui Li | Yawei Luo
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Humans can infer others’ beliefs, intentions, and emotions from observable behavior, an ability known as Theory of Mind (ToM). While current Multimodal Large Language Models (MLLMs) excel at visual understanding, can they identify mental-state cues in videos and reason about evolving mental states? To systematically evaluate spatio-temporal ToM in MLLMs, we introduce ToM-Video, a large-scale benchmark built from diverse real-world narrative videos. Through multi-stage filtering and annotation from massive data sources, ToM-Video provides a high-quality collection of ToM-centric scenes, comprising 1,487 videos and 41K visual question answering (VQA) pairs. We further propose INSPIRE, a post-training framework that, through a carefully designed three-stage curriculum, progressively teaches models to locate ToM-relevant cues and connect them to latent mental-state changes. Experiments show that current MLLMs remain weak at linking observable cues to evolving mental states, while INSPIRE shifts the model from surface description toward compact, evidence-backed inference, leading to higher accuracy and better generalization to external social-reasoning video datasets.