Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval

Minjoon Jung, SeongHo Choi, JooChan Kim, Jin-Hwa Kim, Byoung-Tak Zhang


Abstract
Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query. For narrative videos, e.g., drama or movies, the holistic understanding of temporal dynamics and multimodal reasoning are crucial. Previous works have shown promising results; however, they relied on the expensive query annotations for the VCMR, i.e., the corresponding moment intervals. To overcome this problem, we propose a self-supervised learning framework: Modal-specific Pseudo Query Generation Network (MPGN).First, MPGN selects candidate temporal moments via subtitle-based moment sampling. Then, it generates pseudo queries exploiting both visualand textual information from the selected temporal moments. Through the multimodal information in the pseudo queries, we show that MPGN successfully learns to localize the video corpus moment without any explicit annotation. We validate the effectiveness of MPGN on TVR dataset, showing the competitive results compared with both supervised models and unsupervised setting models.
Anthology ID:
2022.emnlp-main.530
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7769–7781
Language:
URL:
https://aclanthology.org/2022.emnlp-main.530
DOI:
10.18653/v1/2022.emnlp-main.530
Bibkey:
Cite (ACL):
Minjoon Jung, SeongHo Choi, JooChan Kim, Jin-Hwa Kim, and Byoung-Tak Zhang. 2022. Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7769–7781, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval (Jung et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.530.pdf