Visual TTR - Modelling Visual Question Answering in Type Theory with Records

Ronja Utescher


Abstract
In this paper, I will describe a system that was developed for the task of Visual Question Answering. The system uses the rich type universe of Type Theory with Records (TTR) to model both the utterances about the image, the image itself and classifications made related to the two. At its most basic, the decision of whether any given predicate can be assigned to an object in the image is delegated to a CNN. Consequently, images can be judged as evidence for propositions. The end result is a model whose application of perceptual classifiers to a given image is guided by the accompanying utterance.
Anthology ID:
W19-0602
Volume:
Proceedings of the 13th International Conference on Computational Semantics - Student Papers
Month:
May
Year:
2019
Address:
Gothenburg, Sweden
Editors:
Simon Dobnik, Stergios Chatzikyriakidis, Vera Demberg, Kathrein Abu Kwaik, Vladislav Maraev
Venue:
IWCS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
9–14
Language:
URL:
https://aclanthology.org/W19-0602
DOI:
10.18653/v1/W19-0602
Bibkey:
Cite (ACL):
Ronja Utescher. 2019. Visual TTR - Modelling Visual Question Answering in Type Theory with Records. In Proceedings of the 13th International Conference on Computational Semantics - Student Papers, pages 9–14, Gothenburg, Sweden. Association for Computational Linguistics.
Cite (Informal):
Visual TTR - Modelling Visual Question Answering in Type Theory with Records (Utescher, IWCS 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-0602.pdf