Revisiting Hierarchical Text Classification: Inference and Metrics

Roman Plaud, Matthieu Labeau, Antoine Saillenfest, Thomas Bonald


Abstract
Hierarchical text classification (HTC) is the task of assigning labels to a text within a structured space organized as a hierarchy. Recent works treat HTC as a conventional multilabel classification problem, therefore evaluating it as such. We instead propose to evaluate models based on specifically designed hierarchical metrics and we demonstrate the intricacy of metric choice and prediction inference method. We introduce a new challenging dataset and we evaluate fairly, recent sophisticated models, comparing them with a range of simple but strong baselines, including a new theoretically motivated loss. Finally, we show that those baselines are very often competitive with the latest models. This highlights the importance of carefully considering the evaluation methodology when proposing new methods for HTC
Anthology ID:
2024.conll-1.18
Volume:
Proceedings of the 28th Conference on Computational Natural Language Learning
Month:
November
Year:
2024
Address:
Miami, FL, USA
Editors:
Libby Barak, Malihe Alikhani
Venue:
CoNLL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
231–242
Language:
URL:
https://aclanthology.org/2024.conll-1.18
DOI:
Bibkey:
Cite (ACL):
Roman Plaud, Matthieu Labeau, Antoine Saillenfest, and Thomas Bonald. 2024. Revisiting Hierarchical Text Classification: Inference and Metrics. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 231–242, Miami, FL, USA. Association for Computational Linguistics.
Cite (Informal):
Revisiting Hierarchical Text Classification: Inference and Metrics (Plaud et al., CoNLL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.conll-1.18.pdf