A Computational Acquisition Model for Multimodal Word Categorization

Uri Berger, Gabriel Stanovsky, Omri Abend, Lea Frermann


Abstract
Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition, which is believed to rely heavily on cross-modal signals. However, prior studies has been limited by their reliance on vision models trained on large image datasets annotated with a pre-defined set of depicted object categories. This is (a) not faithful to the information children receive and (b) prohibits the evaluation of such models with respect to category learning tasks, due to the pre-imposed category structure. We address this gap, and present a cognitively-inspired, multimodal acquisition model, trained from image-caption pairs on naturalistic data using cross-modal self-supervision. We show that the model learns word categories and object recognition abilities, and presents trends reminiscent of ones reported in the developmental literature.
Anthology ID:
2022.naacl-main.280
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3819–3835
Language:
URL:
https://aclanthology.org/2022.naacl-main.280
DOI:
10.18653/v1/2022.naacl-main.280
Bibkey:
Cite (ACL):
Uri Berger, Gabriel Stanovsky, Omri Abend, and Lea Frermann. 2022. A Computational Acquisition Model for Multimodal Word Categorization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3819–3835, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
A Computational Acquisition Model for Multimodal Word Categorization (Berger et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.280.pdf
Video:
 https://aclanthology.org/2022.naacl-main.280.mp4
Code
 slab-nlp/multimodal_clustering
Data
ImageNetMS COCOVisual Genome