Emilio Raimond
Author directory2026
Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
Timothee Mickus | Claudio Savelli | Eduardo Calò | Emilio Raimond | Stella Frank | Hengyu Luo | Flavio Giobergia | Vincent Segonne | Chuyuan Li | Aman Sinha | Lorenzo Vaiani | Jörg Tiedemann | Raúl Vázquez
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Timothee Mickus | Claudio Savelli | Eduardo Calò | Emilio Raimond | Stella Frank | Hengyu Luo | Flavio Giobergia | Vincent Segonne | Chuyuan Li | Aman Sinha | Lorenzo Vaiani | Jörg Tiedemann | Raúl Vázquez
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
In an age of rapid model turnover, how do we make hallucination evaluation more perennial? We explore whether human-written hallucination samples could take the place of model-generated hallucinations, in order to make benchmarking detection independent of particular models. To this end, we construct SHEEP, a dataset of 1,600 human-written samples, spanning four languages (Chinese, English, French, Italian), and 18,400 samples from five vision-and-language models, all annotated for hallucinations using a fine-grained span-level labeling scheme. We find that human-written samples result in higher agreement and allow greater control of dataset contents, while remaining distributionally similar to samples derived from vision-and-language samples and providing a reasonable portrayal of detection capabilities — suggesting that human data is a viable substitute for model-based hallucination benchmarks.