Kumar Krishna Agrawal
2024
Attribute Diversity Determines the Systematicity Gap in VQA
Ian Berlot-Attwell
|
Kumar Krishna Agrawal
|
Annabelle Michael Carrell
|
Yash Sharma
|
Naomi Saphra
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Although modern neural networks often generalize to new combinations of familiar concepts, the conditions that enable such compositionality have long been an open question. In this work, we study the systematicity gap in visual question answering: the performance difference between reasoning on previously seen and unseen combinations of object attributes. To test, we introduce a novel diagnostic dataset, CLEVR-HOPE. We find that the systematicity gap is not reduced by increasing the quantity of training data, but is reduced by increasing the diversity of training data. In particular, our experiments suggest that the more distinct attribute type combinations are seen during training, the more systematic we can expect the resulting model to be.
2017
Playing with Embeddings : Evaluating embeddings for Robot Language Learning through MUD Games
Anmol Gulati
|
Kumar Krishna Agrawal
Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP
Acquiring language provides a ubiquitous mode of communication, across humans and robots. To this effect, distributional representations of words based on co-occurrence statistics, have provided significant advancements ranging across machine translation to comprehension. In this paper, we study the suitability of using general purpose word-embeddings for language learning in robots. We propose using text-based games as a proxy to evaluating word embedding on real robots. Based in a risk-reward setting, we review the effectiveness of the embeddings in navigating tasks in fantasy games, as an approximation to their performance on more complex scenarios, like language assisted robot navigation.
Search