Human-Model Divergence in the Handling of Vagueness

Elias Stengel-Eskin, Jimena Guallar-Blasco, Benjamin Van Durme


Abstract
While aggregate performance metrics can generate valuable insights at a large scale, their dominance means more complex and nuanced language phenomena, such as vagueness, may be overlooked. Focusing on vague terms (e.g. sunny, cloudy, young, etc.) we inspect the behavior of visually grounded and text-only models, finding systematic divergences from human judgments even when a model’s overall performance is high. To help explain this disparity, we identify two assumptions made by the datasets and models examined and, guided by the philosophy of vagueness, isolate cases where they do not hold.
Anthology ID:
2021.unimplicit-1.6
Volume:
Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language
Month:
August
Year:
2021
Address:
Online
Editors:
Michael Roth, Reut Tsarfaty, Yoav Goldberg
Venue:
unimplicit
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
43–57
Language:
URL:
https://aclanthology.org/2021.unimplicit-1.6
DOI:
10.18653/v1/2021.unimplicit-1.6
Bibkey:
Cite (ACL):
Elias Stengel-Eskin, Jimena Guallar-Blasco, and Benjamin Van Durme. 2021. Human-Model Divergence in the Handling of Vagueness. In Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language, pages 43–57, Online. Association for Computational Linguistics.
Cite (Informal):
Human-Model Divergence in the Handling of Vagueness (Stengel-Eskin et al., unimplicit 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.unimplicit-1.6.pdf