Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks

Steven Derby, Paul Miller, Barry Devereux


Abstract
Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.
Anthology ID:
2020.coling-main.173
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
1906–1921
Language:
URL:
https://aclanthology.org/2020.coling-main.173
DOI:
10.18653/v1/2020.coling-main.173
Bibkey:
Cite (ACL):
Steven Derby, Paul Miller, and Barry Devereux. 2020. Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1906–1921, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Encoding Lexico-Semantic Knowledge using Ensembles of Feature Maps from Deep Convolutional Neural Networks (Derby et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.173.pdf