Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts

Maria Lymperaiou, George Manoliadis, Orfeas Menis Mastromichalakis, Edmund G. Dervakos, Giorgos Stamou


Abstract
Recent breakthroughs in NLP research, such as the advent of Transformer models have indisputably contributed to major advancements in several tasks. However, few works research robustness and explainability issues of their evaluation strategies. In this work, we examine the behavior of high-performing pre-trained language models, focusing on the task of semantic similarity for visual vocabularies. First, we address the need for explainable evaluation metrics, necessary for understanding the conceptual quality of retrieved instances. Our proposed metrics provide valuable insights in local and global level, showcasing the inabilities of widely used approaches. Secondly, adversarial interventions on salient query semantics expose vulnerabilities of opaque metrics and highlight patterns in learned linguistic representations.
Anthology ID:
2022.coling-1.321
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3639–3658
Language:
URL:
https://aclanthology.org/2022.coling-1.321
DOI:
Bibkey:
Cite (ACL):
Maria Lymperaiou, George Manoliadis, Orfeas Menis Mastromichalakis, Edmund G. Dervakos, and Giorgos Stamou. 2022. Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3639–3658, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts (Lymperaiou et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.321.pdf
Data
COCOFlickr30kVisual Genome