PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck

Thang Pham, Peijie Chen, Tin Nguyen, Seunghyun Yoon, Trung Bui, Anh Nguyen


Abstract
CLIP-based classifiers rely on the prompt containing a class name that is known to the text encoder. Therefore, they perform poorly on new classes or the classes whose names rarely appear on the Internet (e.g., scientific names of birds). For fine-grained classification, we propose PEEB – an explainable and editable classifier to (1) express the class name into a set of text descriptors that describe the visual parts of that class; and (2) match the embeddings of the detected parts to their textual descriptors in each class to compute a logit score for classification. In a zero-shot setting where the class names are unknown, PEEB outperforms CLIP by a huge margin (∼10× in top-1 accuracy). Compared to part-based classifiers, PEEB is not only the state-of-the-art (SOTA) on the supervised-learning setting (88.80% and 92.20% accuracy on CUB-200 and Stanford Dogs-120, respectively) but also the first to enable users to edit the text descriptors to form a new classifier without any re-training. Compared to concept bottleneck models, PEEB is also the SOTA in both zero-shot and supervised-learning settings.
Anthology ID:
2024.findings-naacl.131
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2018–2053
Language:
URL:
https://aclanthology.org/2024.findings-naacl.131
DOI:
Bibkey:
Cite (ACL):
Thang Pham, Peijie Chen, Tin Nguyen, Seunghyun Yoon, Trung Bui, and Anh Nguyen. 2024. PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2018–2053, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck (Pham et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.131.pdf
Copyright:
 2024.findings-naacl.131.copyright.pdf