VideoAgent: All-in-One Framework for Video Understanding and Editing

Hengji Zhou, Lingxuan Huang, Jian Wang, Bing Zhou, Si Wu, Lianghao Xia, Chao Huang


Abstract
Video editing has become essential in digital media creation, yet existing automated systems are restricted to short segment processing and domain-specific tasks. They face two critical limitations: i) inability to handle diverse video comprehension and editing operations, and ii) lack of long-video understanding for coherent narrative creation. We propose VideoAgent, an all-in-one agentic framework addressing these challenges through two key innovations. First, we develop automated video shot creation with shot planning agents for coherent narratives and cross-modal retrieval for aligned visual content. Second, we design a multi-agent orchestration framework integrating over thirty specialized editing agents. Intent parsing filters relevant tools while textual-gradient graph optimization assembles complex editing pipelines. Extensive experiments on our newly proposed VideoEdit benchmark and public datasets demonstrate VideoAgent’s superiority over existing multimodal LLMs and agentic systems. VideoAgent achieves 87-95% orchestration success rates while reducing API costs by 60%. Human evaluation across six video categories shows that VideoAgent produces professional-quality content approaching human-level performance, with ratings only 4% below human-created videos.
Anthology ID:
2026.emnlp-main.949
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
21860–21906
Language:
URL:
https://aclanthology.org/2026.emnlp-main.949/
DOI:
Bibkey:
Cite (ACL):
Hengji Zhou, Lingxuan Huang, Jian Wang, Bing Zhou, Si Wu, Lianghao Xia, and Chao Huang. 2026. VideoAgent: All-in-One Framework for Video Understanding and Editing. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 21860–21906, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
VideoAgent: All-in-One Framework for Video Understanding and Editing (Zhou et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.949.pdf