SANDI: Story-and-Images Alignment

Sreyasi Nag Chowdhury, Simon Razniewski, Gerhard Weikum


Abstract
The Internet contains a multitude of social media posts and other of stories where text is interspersed with images. In these contexts, images are not simply used for general illustration, but are judiciously placed in certain spots of a story for multimodal descriptions and narration. In this work we analyze the problem of text-image alignment, and present SANDI, a methodology for automatically selecting images from an image collection and aligning them with text paragraphs of a story. SANDI combines visual tags, user-provided tags and background knowledge, and uses an Integer Linear Program to compute alignments that are semantically meaningful. Experiments show that SANDI can select and align images with texts with high quality of semantic fit.
Anthology ID:
2021.eacl-main.85
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
989–999
Language:
URL:
https://aclanthology.org/2021.eacl-main.85
DOI:
10.18653/v1/2021.eacl-main.85
Bibkey:
Cite (ACL):
Sreyasi Nag Chowdhury, Simon Razniewski, and Gerhard Weikum. 2021. SANDI: Story-and-Images Alignment. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 989–999, Online. Association for Computational Linguistics.
Cite (Informal):
SANDI: Story-and-Images Alignment (Nag Chowdhury et al., EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.85.pdf
Data
ConceptNetMS COCOPlaces205