Exploiting Intrinsic Multilateral Logical Rules for Weakly Supervised Natural Language Video Localization

Zhe Xu, Kun Wei, Xu Yang, Cheng Deng


Abstract
Weakly supervised natural language video localization (WS-NLVL) aims to retrieve the moment corresponding to a language query in a video with only video-language pairs utilized during training. Despite great success, existing WS-NLVL methods seldomly consider the complex temporal relations enclosing the language query (e.g., between the language query and sub-queries decomposed from it or its synonymous query), yielding illogical predictions. In this paper, we propose a novel plug-and-play method, Intrinsic Multilateral Logical Rules, namely IMLR, to exploit intrinsic temporal relations and logical rules for WS-NLVL. Specifically, we formalize queries derived from the original language query as the nodes of a directed graph, i.e., intrinsic temporal relation graph (ITRG), and the temporal relations between them as the edges. Instead of directly prompting a pre-trained language model, a relation-guided prompting method is introduced to generate ITRG in a hierarchical manner. We customize four types of multilateral temporal logical rules (i.e., identity, inclusion, synchronization, and succession) from ITRG and utilize them to train our model. Experiments demonstrate the effectiveness and superiority of our method on the Charades-STA and ActivityNet Captions datasets.
Anthology ID:
2024.acl-long.247
Volume:
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4511–4521
Language:
URL:
https://aclanthology.org/2024.acl-long.247
DOI:
10.18653/v1/2024.acl-long.247
Bibkey:
Cite (ACL):
Zhe Xu, Kun Wei, Xu Yang, and Cheng Deng. 2024. Exploiting Intrinsic Multilateral Logical Rules for Weakly Supervised Natural Language Video Localization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4511–4521, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Exploiting Intrinsic Multilateral Logical Rules for Weakly Supervised Natural Language Video Localization (Xu et al., ACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.acl-long.247.pdf