Do Neural Network Cross-Modal Mappings Really Bridge Modalities?

Guillem Collell, Marie-Francine Moens


Abstract
Feed-forward networks are widely used in cross-modal applications to bridge modalities by mapping distributed vectors of one modality to the other, or to a shared space. The predicted vectors are then used to perform e.g., retrieval or labeling. Thus, the success of the whole system relies on the ability of the mapping to make the neighborhood structure (i.e., the pairwise similarities) of the predicted vectors akin to that of the target vectors. However, whether this is achieved has not been investigated yet. Here, we propose a new similarity measure and two ad hoc experiments to shed light on this issue. In three cross-modal benchmarks we learn a large number of language-to-vision and vision-to-language neural network mappings (up to five layers) using a rich diversity of image and text features and loss functions. Our results reveal that, surprisingly, the neighborhood structure of the predicted vectors consistently resembles more that of the input vectors than that of the target vectors. In a second experiment, we further show that untrained nets do not significantly disrupt the neighborhood (i.e., semantic) structure of the input vectors.
Anthology ID:
P18-2074
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
462–468
Language:
URL:
https://aclanthology.org/P18-2074
DOI:
10.18653/v1/P18-2074
Bibkey:
Cite (ACL):
Guillem Collell and Marie-Francine Moens. 2018. Do Neural Network Cross-Modal Mappings Really Bridge Modalities?. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 462–468, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Do Neural Network Cross-Modal Mappings Really Bridge Modalities? (Collell & Moens, ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-2074.pdf
Note:
 P18-2074.Notes.pdf
Presentation:
 P18-2074.Presentation.pdf
Video:
 https://aclanthology.org/P18-2074.mp4