Enhancing Chinese Multi-Label Text Classification Performance with Response-based Knowledge Distillation

Szu-Chi Huang, Cheng-Fu Cao, Po-Hsun Liao, Lung-Hao Lee, Po-Lei Lee, Kuo-Kai Shyu


Abstract
It’s difficult to optimize individual label performance of multi-label text classification, especially in those imbalanced data containing long-tailed labels. Therefore, this study proposes a response-based knowledge distillation mechanism comprising a teacher model that optimizes binary classifiers of the corresponding labels and a student model that is a standalone multi-label classifier learning from distilled knowledge passed by the teacher model. A total of 2,724 Chinese healthcare texts were collected and manually annotated across nine defined labels, resulting in 8731 labels, each containing an average of 3.2 labels. We used 5-fold cross-validation to compare the performance of several multi-label models, including TextRNN, TextCNN, HAN, and GRU-att. Experimental results indicate that using the proposed knowledge distillation mechanism effectively improved the performance no matter which model was used, about 2-3% of micro-F1, 4-6% of macro-F1, 3-4% of weighted-F1 and 1-2% of subset accuracy for performance enhancement.
Anthology ID:
2022.rocling-1.4
Volume:
Proceedings of the 34th Conference on Computational Linguistics and Speech Processing (ROCLING 2022)
Month:
November
Year:
2022
Address:
Taipei, Taiwan
Editors:
Yung-Chun Chang, Yi-Chin Huang
Venue:
ROCLING
SIG:
Publisher:
The Association for Computational Linguistics and Chinese Language Processing (ACLCLP)
Note:
Pages:
25–31
Language:
Chinese
URL:
https://aclanthology.org/2022.rocling-1.4
DOI:
Bibkey:
Cite (ACL):
Szu-Chi Huang, Cheng-Fu Cao, Po-Hsun Liao, Lung-Hao Lee, Po-Lei Lee, and Kuo-Kai Shyu. 2022. Enhancing Chinese Multi-Label Text Classification Performance with Response-based Knowledge Distillation. In Proceedings of the 34th Conference on Computational Linguistics and Speech Processing (ROCLING 2022), pages 25–31, Taipei, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP).
Cite (Informal):
Enhancing Chinese Multi-Label Text Classification Performance with Response-based Knowledge Distillation (Huang et al., ROCLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.rocling-1.4.pdf