Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos

Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou


Abstract
We address the problem of temporal sentence localization in videos (TSLV). Traditional methods follow a top-down framework which localizes the target segment with pre-defined segment proposals. Although they have achieved decent performance, the proposals are handcrafted and redundant. Recently, bottom-up framework attracts increasing attention due to its superior efficiency. It directly predicts the probabilities for each frame as a boundary. However, the performance of bottom-up model is inferior to the top-down counterpart as it fails to exploit the segment-level interaction. In this paper, we propose an Adaptive Proposal Generation Network (APGN) to maintain the segment-level interaction while speeding up the efficiency. Specifically, we first perform a foreground-background classification upon the video and regress on the foreground frames to adaptively generate proposals. In this way, the handcrafted proposal design is discarded and the redundant proposals are decreased. Then, a proposal consolidation module is further developed to enhance the semantics of the generated proposals. Finally, we locate the target moments with these generated proposals following the top-down framework. Extensive experiments show that our proposed APGN significantly outperforms previous state-of-the-art methods on three challenging benchmarks.
Anthology ID:
2021.emnlp-main.732
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9292–9301
Language:
URL:
https://aclanthology.org/2021.emnlp-main.732
DOI:
10.18653/v1/2021.emnlp-main.732
Bibkey:
Cite (ACL):
Daizong Liu, Xiaoye Qu, Jianfeng Dong, and Pan Zhou. 2021. Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9292–9301, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos (Liu et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.732.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.732.mp4
Data
ActivityNet CaptionsCharades-STA