Learning Label Hierarchy with Supervised Contrastive Learning

Ruixue Lian, William Sethares, Junjie Hu


Abstract
Supervised contrastive learning (SCL) frameworks treat each class as independent and thus consider all classes to be equally important. This neglects the common scenario in which label hierarchy exists, where fine-grained classes under the same category show more similarity than very different ones. This paper introduces a family of Label-Aware SCL methods (LA-SCL) that incorporates hierarchical information to SCL by leveraging similarities between classes, resulting in creating a more well-structured and discriminative feature space. This is achieved by first adjusting the distance between instances based on measures of the proximity of their classes with the scaled instance-instance-wise contrastive. An additional instance-center-wise contrastive is introduced to move within-class examples closer to their centers, which are represented by a set of learnable label parameters. The learned label parameters can be directly used as a nearest neighbor classifier without further finetuning. In this way, a better feature representation is generated with improvements of intra-cluster compactness and inter-cluster separation. Experiments on three datasets show that the proposed LA-SCL works well on text classification of distinguishing a single label among multi-labels, outperforming the baseline supervised approaches. Our code is publicly available 1.
Anthology ID:
2024.findings-eacl.108
Volume:
Findings of the Association for Computational Linguistics: EACL 2024
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1569–1581
Language:
URL:
https://aclanthology.org/2024.findings-eacl.108
DOI:
Bibkey:
Cite (ACL):
Ruixue Lian, William Sethares, and Junjie Hu. 2024. Learning Label Hierarchy with Supervised Contrastive Learning. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1569–1581, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Learning Label Hierarchy with Supervised Contrastive Learning (Lian et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-eacl.108.pdf
Video:
 https://aclanthology.org/2024.findings-eacl.108.mp4