“Caption” as a Coherence Relation: Evidence and Implications

Malihe Alikhani, Matthew Stone


Abstract
We study verbs in image–text corpora, contrasting caption corpora, where texts are explicitly written to characterize image content, with depiction corpora, where texts and images may stand in more general relations. Captions show a distinctively limited distribution of verbs, with strong preferences for specific tense, aspect, lexical aspect, and semantic field. These limitations, which appear in data elicited by a range of methods, restrict the utility of caption corpora to inform image retrieval, multimodal document generation, and perceptually-grounded semantic models. We suggest that these limitations reflect the discourse constraints in play when subjects write texts to accompany imagery, so we argue that future development of image–text corpora should work to increase the diversity of event descriptions, while looking explicitly at the different ways text and imagery can be coherently related.
Anthology ID:
W19-1806
Volume:
Proceedings of the Second Workshop on Shortcomings in Vision and Language
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Venues:
NAACL | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
58–67
Language:
URL:
https://aclanthology.org/W19-1806
DOI:
10.18653/v1/W19-1806
Bibkey:
Cite (ACL):
Malihe Alikhani and Matthew Stone. 2019. “Caption” as a Coherence Relation: Evidence and Implications. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 58–67, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
“Caption” as a Coherence Relation: Evidence and Implications (Alikhani & Stone, 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-1806.pdf
Data
COCOFlickr30kVIST