@inproceedings{zaranis-etal-2026-well,
title = "How Well Do Vision-Language Models Understand Movies?",
author = "Zaranis, Emmanouil and
Farinhas, Ant{\'o}nio and
Santos, Saul and
Canaverde, Beatriz and
Ramos, Miguel Moura and
Mohammed, Wafaa and
Attanasio, Giuseppe and
Zerva, Chrysoula and
Sivakumaran, Nithin and
Yu, Shoubin and
Bueno-Benito, Elena and
Surikuchi, Aditya K and
Peters, Ben and
Sanchez Villegas, Danae and
Viveiros, Andre G. and
Vasylenko, Pavlo and
Liao, Baohao and
Sannigrahi, Sonal and
Yoon, Jaehong and
Stengel-Eskin, Elias and
Dimiccoli, Mariella and
Lanz, Oswald and
Suglia, Alessandro and
Bansal, Mohit and
Pezzelle, Sandro and
Frank, Stella and
Niculae, Vlad and
Elliott, Desmond and
Bernardi, Raffaella and
Fern{\'a}ndez, Raquel and
Martins, Andre",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1702/",
pages = "38762--38801",
ISBN = "979-8-89176-459-0",
abstract = "Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos{---}but can they understand the narrative structure of movies as humans do? To investigate this question, we introduce ${MF}^2$ ($\textit{Movie Facts and Fibs}$), a fully manually curated benchmark designed to evaluate whether models can identify, recall, and comprehend information central to the movie{'}s storyline, using both the visual and textual modalities. ${MF}^2$ includes over 50 $\textit{open-licensed}$ movies, each paired with manually constructed sets of claim pairs: one true claim ($\textit{fact}$) and one plausible but false ($\textit{fib}$), totaling over 850 pairs. Our analysis across 20 open-source and 2 proprietary models reveals a substantial gap relative to human performance. While subtitles provide a strong signal for narrative understanding, other textual sources (e.g., synopses) are less informative. In contrast, when visual information is distributed across the entire movie and jointly embedded with subtitles, models struggle to identify and effectively use the relevant content. However, when the visual input is restricted to claim-relevant visual content, models improve significantly, indicating substantial room for improvement in processing large multimodal contexts and distinguishing relevant from irrelevant information. Overall, our findings highlight persistent challenges in multimodal narrative understanding for current VLMs. We release our data and code to support further research in this direction."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zaranis-etal-2026-well">
<titleInfo>
<title>How Well Do Vision-Language Models Understand Movies?</title>
</titleInfo>
<name type="personal">
<namePart type="given">Emmanouil</namePart>
<namePart type="family">Zaranis</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">António</namePart>
<namePart type="family">Farinhas</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Saul</namePart>
<namePart type="family">Santos</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Beatriz</namePart>
<namePart type="family">Canaverde</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Miguel</namePart>
<namePart type="given">Moura</namePart>
<namePart type="family">Ramos</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Wafaa</namePart>
<namePart type="family">Mohammed</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Giuseppe</namePart>
<namePart type="family">Attanasio</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chrysoula</namePart>
<namePart type="family">Zerva</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nithin</namePart>
<namePart type="family">Sivakumaran</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shoubin</namePart>
<namePart type="family">Yu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Elena</namePart>
<namePart type="family">Bueno-Benito</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Aditya</namePart>
<namePart type="given">K</namePart>
<namePart type="family">Surikuchi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ben</namePart>
<namePart type="family">Peters</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Danae</namePart>
<namePart type="family">Sanchez Villegas</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Andre</namePart>
<namePart type="given">G</namePart>
<namePart type="family">Viveiros</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Pavlo</namePart>
<namePart type="family">Vasylenko</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Baohao</namePart>
<namePart type="family">Liao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sonal</namePart>
<namePart type="family">Sannigrahi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jaehong</namePart>
<namePart type="family">Yoon</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Elias</namePart>
<namePart type="family">Stengel-Eskin</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Mariella</namePart>
<namePart type="family">Dimiccoli</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Oswald</namePart>
<namePart type="family">Lanz</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Alessandro</namePart>
<namePart type="family">Suglia</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Mohit</namePart>
<namePart type="family">Bansal</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sandro</namePart>
<namePart type="family">Pezzelle</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Stella</namePart>
<namePart type="family">Frank</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Vlad</namePart>
<namePart type="family">Niculae</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Raffaella</namePart>
<namePart type="family">Bernardi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Raquel</namePart>
<namePart type="family">Fernández</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Andre</namePart>
<namePart type="family">Martins</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos—but can they understand the narrative structure of movies as humans do? To investigate this question, we introduce MF² (Movie Facts and Fibs), a fully manually curated benchmark designed to evaluate whether models can identify, recall, and comprehend information central to the movie’s storyline, using both the visual and textual modalities. MF² includes over 50 open-licensed movies, each paired with manually constructed sets of claim pairs: one true claim (fact) and one plausible but false (fib), totaling over 850 pairs. Our analysis across 20 open-source and 2 proprietary models reveals a substantial gap relative to human performance. While subtitles provide a strong signal for narrative understanding, other textual sources (e.g., synopses) are less informative. In contrast, when visual information is distributed across the entire movie and jointly embedded with subtitles, models struggle to identify and effectively use the relevant content. However, when the visual input is restricted to claim-relevant visual content, models improve significantly, indicating substantial room for improvement in processing large multimodal contexts and distinguishing relevant from irrelevant information. Overall, our findings highlight persistent challenges in multimodal narrative understanding for current VLMs. We release our data and code to support further research in this direction.</abstract>
<identifier type="citekey">zaranis-etal-2026-well</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1702/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>38762</start>
<end>38801</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T How Well Do Vision-Language Models Understand Movies?
%A Zaranis, Emmanouil
%A Farinhas, António
%A Santos, Saul
%A Canaverde, Beatriz
%A Ramos, Miguel Moura
%A Mohammed, Wafaa
%A Attanasio, Giuseppe
%A Zerva, Chrysoula
%A Sivakumaran, Nithin
%A Yu, Shoubin
%A Bueno-Benito, Elena
%A Surikuchi, Aditya K.
%A Peters, Ben
%A Sanchez Villegas, Danae
%A Viveiros, Andre G.
%A Vasylenko, Pavlo
%A Liao, Baohao
%A Sannigrahi, Sonal
%A Yoon, Jaehong
%A Stengel-Eskin, Elias
%A Dimiccoli, Mariella
%A Lanz, Oswald
%A Suglia, Alessandro
%A Bansal, Mohit
%A Pezzelle, Sandro
%A Frank, Stella
%A Niculae, Vlad
%A Elliott, Desmond
%A Bernardi, Raffaella
%A Fernández, Raquel
%A Martins, Andre
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F zaranis-etal-2026-well
%X Vision-language models (VLMs) can increasingly handle extended contexts and answer questions about long videos—but can they understand the narrative structure of movies as humans do? To investigate this question, we introduce MF² (Movie Facts and Fibs), a fully manually curated benchmark designed to evaluate whether models can identify, recall, and comprehend information central to the movie’s storyline, using both the visual and textual modalities. MF² includes over 50 open-licensed movies, each paired with manually constructed sets of claim pairs: one true claim (fact) and one plausible but false (fib), totaling over 850 pairs. Our analysis across 20 open-source and 2 proprietary models reveals a substantial gap relative to human performance. While subtitles provide a strong signal for narrative understanding, other textual sources (e.g., synopses) are less informative. In contrast, when visual information is distributed across the entire movie and jointly embedded with subtitles, models struggle to identify and effectively use the relevant content. However, when the visual input is restricted to claim-relevant visual content, models improve significantly, indicating substantial room for improvement in processing large multimodal contexts and distinguishing relevant from irrelevant information. Overall, our findings highlight persistent challenges in multimodal narrative understanding for current VLMs. We release our data and code to support further research in this direction.
%U https://aclanthology.org/2026.emnlp-main.1702/
%P 38762-38801
Markdown (Informal)
[How Well Do Vision-Language Models Understand Movies?](https://aclanthology.org/2026.emnlp-main.1702/) (Zaranis et al., EMNLP 2026)
ACL
- Emmanouil Zaranis, António Farinhas, Saul Santos, Beatriz Canaverde, Miguel Moura Ramos, Wafaa Mohammed, Giuseppe Attanasio, Chrysoula Zerva, Nithin Sivakumaran, Shoubin Yu, Elena Bueno-Benito, Aditya K Surikuchi, Ben Peters, Danae Sanchez Villegas, Andre G. Viveiros, Pavlo Vasylenko, Baohao Liao, Sonal Sannigrahi, Jaehong Yoon, Elias Stengel-Eskin, Mariella Dimiccoli, Oswald Lanz, Alessandro Suglia, Mohit Bansal, Sandro Pezzelle, Stella Frank, Vlad Niculae, Desmond Elliott, Raffaella Bernardi, Raquel Fernández, and Andre Martins. 2026. How Well Do Vision-Language Models Understand Movies?. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 38762–38801, Budapest, Hungary. Association for Computational Linguistics.