There’s a Time and Place for Reasoning Beyond the Image

Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Vondrick, Dan Roth


Abstract
Images are often more significant than only the pixels to human eyes, as we can infer, associate, and reason with contextual information from other sources to establish a more complete picture. For example, in Figure 1, we can find a way to identify the news articles related to the picture through segment-wise understandings of the signs, the buildings, the crowds, and more. This reasoning could provide the time and place the image was taken, which will help us in subsequent tasks, such as automatic storyline construction, correction of image source in intended effect photographs, and upper-stream processing such as image clustering for certain location or time. In this work, we formulate this problem and introduce TARA: a dataset with 16k images with their associated news, time, and location, automatically extracted from New York Times, and an additional 61k examples as distant supervision from WIT. On top of the extractions, we present a crowdsourced subset in which we believe it is possible to find the images’ spatio-temporal information for evaluation purpose. We show that there exists a 70% gap between a state-of-the-art joint model and human performance, which is slightly filled by our proposed model that uses segment-wise reasoning, motivating higher-level vision-language joint models that can conduct open-ended reasoning with world knowledge. The data and code are publicly available at https://github.com/zeyofu/TARA.
Anthology ID:
2022.acl-long.81
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1138–1149
Language:
URL:
https://aclanthology.org/2022.acl-long.81
DOI:
10.18653/v1/2022.acl-long.81
Bibkey:
Cite (ACL):
Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Vondrick, and Dan Roth. 2022. There’s a Time and Place for Reasoning Beyond the Image. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1138–1149, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
There’s a Time and Place for Reasoning Beyond the Image (Fu et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.81.pdf
Video:
 https://aclanthology.org/2022.acl-long.81.mp4
Code
 zeyofu/tara
Data
WIT