Video Captioning with Multi-Faceted Attention

Xiang Long, Chuang Gan, Gerard de Melo


Abstract
Video captioning has attracted an increasing amount of interest, due in part to its potential for improved accessibility and information retrieval. While existing methods rely on different kinds of visual features and model architectures, they do not make full use of pertinent semantic cues. We present a unified and extensible framework to jointly leverage multiple sorts of visual features and semantic attributes. Our novel architecture builds on LSTMs with two multi-faceted attention layers. These first learn to automatically select the most salient visual features or semantic attributes, and then yield overall representations for the input and output of the sentence generation component via custom feature scaling operations. Experimental results on the challenging MSVD and MSR-VTT datasets show that our framework outperforms previous work and performs robustly even in the presence of added noise to the features and attributes.
Anthology ID:
Q18-1013
Volume:
Transactions of the Association for Computational Linguistics, Volume 6
Month:
Year:
2018
Address:
Cambridge, MA
Editors:
Lillian Lee, Mark Johnson, Kristina Toutanova, Brian Roark
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
173–184
Language:
URL:
https://aclanthology.org/Q18-1013
DOI:
10.1162/tacl_a_00013
Bibkey:
Cite (ACL):
Xiang Long, Chuang Gan, and Gerard de Melo. 2018. Video Captioning with Multi-Faceted Attention. Transactions of the Association for Computational Linguistics, 6:173–184.
Cite (Informal):
Video Captioning with Multi-Faceted Attention (Long et al., TACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/Q18-1013.pdf
Data
MSVD