Geo-Aware Image Caption Generation

Sofia Nikiforova, Tejaswini Deoskar, Denis Paperno, Yoad Winter


Abstract
Standard image caption generation systems produce generic descriptions of images and do not utilize any contextual information or world knowledge. In particular, they are unable to generate captions that contain references to the geographic context of an image, for example, the location where a photograph is taken or relevant geographic objects around an image location. In this paper, we develop a geo-aware image caption generation system, which incorporates geographic contextual information into a standard image captioning pipeline. We propose a way to build an image-specific representation of the geographic context and adapt the caption generation network to produce appropriate geographic names in the image descriptions. We evaluate our system on a novel captioning dataset that contains contextualized captions and geographic metadata and achieve substantial improvements in BLEU, ROUGE, METEOR and CIDEr scores. We also introduce a new metric to assess generated geographic references directly and empirically demonstrate our system’s ability to produce captions with relevant and factually accurate geographic referencing.
Anthology ID:
2020.coling-main.280
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3143–3156
Language:
URL:
https://aclanthology.org/2020.coling-main.280
DOI:
10.18653/v1/2020.coling-main.280
Bibkey:
Cite (ACL):
Sofia Nikiforova, Tejaswini Deoskar, Denis Paperno, and Yoad Winter. 2020. Geo-Aware Image Caption Generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3143–3156, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Geo-Aware Image Caption Generation (Nikiforova et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.280.pdf
Data
COCO