Natural Language Semantics With Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics

David Schlangen


Abstract
Propelling, and propelled by, the “deep learning revolution”, recent years have seen the introduction of ever larger corpora of images annotated with natural language expressions. We survey some of these corpora, taking a perspective that reverses the usual directionality, as it were, by viewing the images as semantic annotation of the natural language expressions. We discuss datasets that can be derived from the corpora, and tasks of potential interest for computational semanticists that can be defined on those. In this, we make use of relations provided by the corpora (namely, the link between expression and image, and that between two expressions linked to the same image) and relations that we can add (similarity relations between expressions, or between images). Specifically, we show that in this way we can create data that can be used to learn and evaluate lexical and compositional grounded semantics, and we show that the “linked to same image” relation tracks a semantic implication relation that is recognisable to annotators even in the absence of the linking image as evidence. Finally, as an example of possible benefits of this approach, we show that an exemplar-model-based approach to implication beats a (simple) distributional space-based one on some derived datasets, while lending itself to explainability.
Anthology ID:
W19-0424
Volume:
Proceedings of the 13th International Conference on Computational Semantics - Long Papers
Month:
May
Year:
2019
Address:
Gothenburg, Sweden
Venues:
IWCS | WS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
283–294
Language:
URL:
https://aclanthology.org/W19-0424
DOI:
10.18653/v1/W19-0424
Bibkey:
Cite (ACL):
David Schlangen. 2019. Natural Language Semantics With Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 283–294, Gothenburg, Sweden. Association for Computational Linguistics.
Cite (Informal):
Natural Language Semantics With Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics (Schlangen, 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-0424.pdf