WhyAct: Identifying Action Reasons in Lifestyle Vlogs

Oana Ignat, Santiago Castro, Hanwen Miao, Weiji Li, Rada Mihalcea


Abstract
We aim to automatically identify human action reasons in online videos. We focus on the widespread genre of lifestyle vlogs, in which people perform actions while verbally describing them. We introduce and make publicly available the WhyAct dataset, consisting of 1,077 visual actions manually annotated with their reasons. We describe a multimodal model that leverages visual and textual information to automatically infer the reasons corresponding to an action presented in the video.
Anthology ID:
2021.emnlp-main.392
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4770–4785
Language:
URL:
https://aclanthology.org/2021.emnlp-main.392
DOI:
10.18653/v1/2021.emnlp-main.392
Bibkey:
Cite (ACL):
Oana Ignat, Santiago Castro, Hanwen Miao, Weiji Li, and Rada Mihalcea. 2021. WhyAct: Identifying Action Reasons in Lifestyle Vlogs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4770–4785, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
WhyAct: Identifying Action Reasons in Lifestyle Vlogs (Ignat et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.392.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.392.mp4
Code
 michigannlp/vlog_action_reason
Data
WhyActConceptNetIfAct