Memory-efficient Temporal Moment Localization in Long Videos

Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Basura Fernando, Hiroya Takamura, Qi Wu


Abstract
Temporal Moment Localization is a challenging multi-modal task which aims to identify the start and end timestamps of a moment of interest in an input untrimmed video, given a query in natural language. Solving this task correctly requires understanding the temporal relationships in the entire input video, but processing such long inputs and reasoning about them is memory and computationally expensive. In light of this issue, we propose Stochastic Bucket-wise Feature Sampling (SBFS), a stochastic sampling module that allows methods to process long videos at a constant memory footprint. We further combine SBFS with a new consistency loss to propose Locformer, a Transformer-based model that can process videos as long as 18 minutes. We test our proposals on relevant benchmark datasets, showing that not only can Locformer achieve excellent results, but also that our sampling is more effective than competing counterparts. Concretely, SBFS consistently improves the performance of prior work, by up to 3.13% in the mean temporal IoU, leading to a new state-of-the-art performance on Charades-STA and YouCookII, while also obtaining up to 12.8x speed-up at testing time and reducing memory requirements by up to 5x.
Anthology ID:
2023.eacl-main.140
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1909–1924
Language:
URL:
https://aclanthology.org/2023.eacl-main.140
DOI:
10.18653/v1/2023.eacl-main.140
Bibkey:
Cite (ACL):
Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Basura Fernando, Hiroya Takamura, and Qi Wu. 2023. Memory-efficient Temporal Moment Localization in Long Videos. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1909–1924, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Memory-efficient Temporal Moment Localization in Long Videos (Rodriguez-Opazo et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.140.pdf
Video:
 https://aclanthology.org/2023.eacl-main.140.mp4