How agents see things: On visual representations in an emergent language game

Diane Bouchacourt, Marco Baroni


Abstract
There is growing interest in the language developed by agents interacting in emergent-communication settings. Earlier studies have focused on the agents’ symbol usage, rather than on their representation of visual input. In this paper, we consider the referential games of Lazaridou et al. (2017), and investigate the representations the agents develop during their evolving interaction. We find that the agents establish successful communication by inducing visual representations that almost perfectly align with each other, but, surprisingly, do not capture the conceptual properties of the objects depicted in the input images. We conclude that, if we care about developing language-like communication systems, we must pay more attention to the visual semantics agents associate to the symbols they use.
Anthology ID:
D18-1119
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
981–985
Language:
URL:
https://aclanthology.org/D18-1119
DOI:
10.18653/v1/D18-1119
Bibkey:
Cite (ACL):
Diane Bouchacourt and Marco Baroni. 2018. How agents see things: On visual representations in an emergent language game. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 981–985, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
How agents see things: On visual representations in an emergent language game (Bouchacourt & Baroni, EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1119.pdf
Video:
 https://aclanthology.org/D18-1119.mp4
Data
ImageNet