Would you still call this Dax? Novel Visual References in VLMs and Humans

Ada Defne Tur, Gaurav Kamath, Joyce Chai, Siva Reddy, Benno Krojer


Abstract
Vision-language models (VLMs), like human learners, are frequently exposed to new visual concepts, but how they map novel visual references to language after exposure remains largely underexplored, particularly when those references contradict prior knowledge from pre-training. To study this, we present the Novel Visual References Dataset (NVRD): 19,176 images spanning 90 visual concepts across different levels of visual novelty, each with up to 20 increasingly perturbed versions of the original object to probe generalization. Unlike prior work on visual augmentations of familiar concepts, NVRD comprises entirely novel, open-ended stimuli constructed from scratch, mirroring how humans encounter genuinely new concepts. We evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments for direct human–model comparison, and find that (i) models struggle to acquire novel concepts in-context when they contradict prior knowledge, and (ii) while models and humans show correlated sensitivity to visual perturbations, models significantly overgeneralize, extending learned labels to stimuli that humans reject. We contribute NVRD as a corpus and benchmark for research on visual concept learning in both humans and machines.
Anthology ID:
2026.emnlp-main.700
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
16148–16183
Language:
URL:
https://aclanthology.org/2026.emnlp-main.700/
DOI:
Bibkey:
Cite (ACL):
Ada Defne Tur, Gaurav Kamath, Joyce Chai, Siva Reddy, and Benno Krojer. 2026. Would you still call this Dax? Novel Visual References in VLMs and Humans. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 16148–16183, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Would you still call this Dax? Novel Visual References in VLMs and Humans (Tur et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.700.pdf