mTVR: Multilingual Moment Retrieval in Videos

Jie Lei, Tamara Berg, Mohit Bansal


Abstract
We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21.8K TV show video clips. The dataset is collected by extending the popular TVR dataset (in English) with paired Chinese queries and subtitles. Compared to existing moment retrieval datasets, mTVR is multilingual, larger, and comes with diverse annotations. We further propose mXML, a multilingual moment retrieval model that learns and operates on data from both languages, via encoder parameter sharing and language neighborhood constraints. We demonstrate the effectiveness of mXML on the newly collected mTVR dataset, where mXML outperforms strong monolingual baselines while using fewer parameters. In addition, we also provide detailed dataset analyses and model ablations. Data and code are publicly available at https://github.com/jayleicn/mTVRetrieval
Anthology ID:
2021.acl-short.92
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
726–734
Language:
URL:
https://aclanthology.org/2021.acl-short.92
DOI:
10.18653/v1/2021.acl-short.92
Bibkey:
Cite (ACL):
Jie Lei, Tamara Berg, and Mohit Bansal. 2021. mTVR: Multilingual Moment Retrieval in Videos. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 726–734, Online. Association for Computational Linguistics.
Cite (Informal):
mTVR: Multilingual Moment Retrieval in Videos (Lei et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-short.92.pdf
Video:
 https://aclanthology.org/2021.acl-short.92.mp4
Code
 jayleicn/mTVRetrieval
Data
mTVRTVR