Does language help generalization in vision models?

Benjamin Devillers, Bhavin Choksi, Romain Bielawski, Rufin VanRullen


Abstract
Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This could imply that linguistic or “semantic grounding” confers additional generalization abilities to the visual feature space. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models.
Anthology ID:
2021.conll-1.13
Volume:
Proceedings of the 25th Conference on Computational Natural Language Learning
Month:
November
Year:
2021
Address:
Online
Venues:
CoNLL | EMNLP
SIG:
SIGNLL
Publisher:
Association for Computational Linguistics
Note:
Pages:
171–182
Language:
URL:
https://aclanthology.org/2021.conll-1.13
DOI:
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.conll-1.13.pdf
Code
 bdvllrs/generalization-vision
Data
CIFAR-10CIFAR-100COCOCUB-200-2011Fashion-MNISTHowTo100MMNISTSVHN