@inproceedings{he-etal-2026-vgi,
title = "{VGI}-Bench: Probing Visual Intelligence in Video Generation Models",
author = "He, Xuan and
Wei, Cong and
Cheng, Yuhao and
Ma, Linrui and
Zhang, Yuxuan and
Li, Zuojun and
Wen, Yuhao and
Jiang, Jize and
Liu, Zeyi and
Hao, Yuren and
Cai, Songcheng and
Wu, Keming and
Du, Penghui and
Zou, Kai and
Yang, Rui and
Sun, Chenkai and
Yang, Ke and
Nie, Ping and
Allen, Kelsey R and
Wang, Chenglong and
Galley, Michel and
Gao, Jianfeng and
Zhai, ChengXiang",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1770/",
pages = "40335--40371",
ISBN = "979-8-89176-459-0",
abstract = "Recent studies suggest that video generative models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-Bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of domains and skill tags for evaluating the visual reasoning capabilities of video generative models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0{\%} under our evaluation criteria. Our findings further reveal the input sensitivity, skill-bounded transfer from synthetic fine-tuning, and limited denoising-time self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-Bench will help stimulate the development of next-generation video generative models. Data and code are available at [our project page](https://hexuan21.github.io/VGI-Bench/)."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="he-etal-2026-vgi">
<titleInfo>
<title>VGI-Bench: Probing Visual Intelligence in Video Generation Models</title>
</titleInfo>
<name type="personal">
<namePart type="given">Xuan</namePart>
<namePart type="family">He</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Cong</namePart>
<namePart type="family">Wei</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuhao</namePart>
<namePart type="family">Cheng</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Linrui</namePart>
<namePart type="family">Ma</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuxuan</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zuojun</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuhao</namePart>
<namePart type="family">Wen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jize</namePart>
<namePart type="family">Jiang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zeyi</namePart>
<namePart type="family">Liu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuren</namePart>
<namePart type="family">Hao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Songcheng</namePart>
<namePart type="family">Cai</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Keming</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Penghui</namePart>
<namePart type="family">Du</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kai</namePart>
<namePart type="family">Zou</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Rui</namePart>
<namePart type="family">Yang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chenkai</namePart>
<namePart type="family">Sun</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ke</namePart>
<namePart type="family">Yang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ping</namePart>
<namePart type="family">Nie</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Kelsey</namePart>
<namePart type="given">R</namePart>
<namePart type="family">Allen</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chenglong</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Michel</namePart>
<namePart type="family">Galley</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jianfeng</namePart>
<namePart type="family">Gao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">ChengXiang</namePart>
<namePart type="family">Zhai</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Recent studies suggest that video generative models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-Bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of domains and skill tags for evaluating the visual reasoning capabilities of video generative models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our findings further reveal the input sensitivity, skill-bounded transfer from synthetic fine-tuning, and limited denoising-time self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-Bench will help stimulate the development of next-generation video generative models. Data and code are available at [our project page](https://hexuan21.github.io/VGI-Bench/).</abstract>
<identifier type="citekey">he-etal-2026-vgi</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1770/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>40335</start>
<end>40371</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T VGI-Bench: Probing Visual Intelligence in Video Generation Models
%A He, Xuan
%A Wei, Cong
%A Cheng, Yuhao
%A Ma, Linrui
%A Zhang, Yuxuan
%A Li, Zuojun
%A Wen, Yuhao
%A Jiang, Jize
%A Liu, Zeyi
%A Hao, Yuren
%A Cai, Songcheng
%A Wu, Keming
%A Du, Penghui
%A Zou, Kai
%A Yang, Rui
%A Sun, Chenkai
%A Yang, Ke
%A Nie, Ping
%A Allen, Kelsey R.
%A Wang, Chenglong
%A Galley, Michel
%A Gao, Jianfeng
%A Zhai, ChengXiang
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F he-etal-2026-vgi
%X Recent studies suggest that video generative models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-Bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of domains and skill tags for evaluating the visual reasoning capabilities of video generative models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our findings further reveal the input sensitivity, skill-bounded transfer from synthetic fine-tuning, and limited denoising-time self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-Bench will help stimulate the development of next-generation video generative models. Data and code are available at [our project page](https://hexuan21.github.io/VGI-Bench/).
%U https://aclanthology.org/2026.emnlp-main.1770/
%P 40335-40371
Markdown (Informal)
[VGI-Bench: Probing Visual Intelligence in Video Generation Models](https://aclanthology.org/2026.emnlp-main.1770/) (He et al., EMNLP 2026)
ACL
- Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang, Zeyi Liu, Yuren Hao, Songcheng Cai, Keming Wu, Penghui Du, Kai Zou, Rui Yang, Chenkai Sun, Ke Yang, Ping Nie, Kelsey R Allen, Chenglong Wang, Michel Galley, Jianfeng Gao, and ChengXiang Zhai. 2026. VGI-Bench: Probing Visual Intelligence in Video Generation Models. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 40335–40371, Budapest, Hungary. Association for Computational Linguistics.