On the Role of Scene Graphs in Image Captioning

Dalin Wang, Daniel Beck, Trevor Cohn


Abstract
Scene graphs represent semantic information in images, which can help image captioning system to produce more descriptive outputs versus using only the image as context. Recent captioning approaches rely on ad-hoc approaches to obtain graphs for images. However, those graphs introduce noise and it is unclear the effect of parser errors on captioning accuracy. In this work, we investigate to what extent scene graphs can help image captioning. Our results show that a state-of-the-art scene graph parser can boost performance almost as much as the ground truth graphs, showing that the bottleneck currently resides more on the captioning models than on the performance of the scene graph parser.
Anthology ID:
D19-6405
Volume:
Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Aditya Mogadala, Dietrich Klakow, Sandro Pezzelle, Marie-Francine Moens
Venue:
WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
29–34
Language:
URL:
https://aclanthology.org/D19-6405
DOI:
10.18653/v1/D19-6405
Bibkey:
Cite (ACL):
Dalin Wang, Daniel Beck, and Trevor Cohn. 2019. On the Role of Scene Graphs in Image Captioning. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN), pages 29–34, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
On the Role of Scene Graphs in Image Captioning (Wang et al., 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-6405.pdf