Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification
Tetsuya
Sakai
author
2021-08
text
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Chengqing
Zong
editor
Fei
Xia
editor
Wenjie
Li
editor
Roberto
Navigli
editor
Association for Computational Linguistics
Online
conference publication
Ordinal Classification (OC) is an important classification task where the classes are ordinal. For example, an OC task for sentiment analysis could have the following classes: highly positive, positive, neutral, negative, highly negative. Clearly, evaluation measures for an OC task should penalise misclassifications by considering the ordinal nature of the classes. Ordinal Quantification (OQ) is a related task where the gold data is a distribution over ordinal classes, and the system is required to estimate this distribution. Evaluation measures for an OQ task should also take the ordinal nature of the classes into account. However, for both OC and OQ, there are only a small number of known evaluation measures that meet this basic requirement. In the present study, we utilise data from the SemEval and NTCIR communities to clarify the properties of nine evaluation measures in the context of OC tasks, and six measures in the context of OQ tasks.
sakai-2021-evaluating
10.18653/v1/2021.acl-long.214
https://aclanthology.org/2021.acl-long.214
2021-08
2759
2769