Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation

Hiroaki Funayama, Shota Sasaki, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, Masato Mita, Kentaro Inui


Abstract
Many recent Short Answer Scoring (SAS) systems have employed Quadratic Weighted Kappa (QWK) as the evaluation measure of their systems. However, we hypothesize that QWK is unsatisfactory for the evaluation of the SAS systems when we consider measuring their effectiveness in actual usage. We introduce a new task formulation of SAS that matches the actual usage. In our formulation, the SAS systems should extract as many scoring predictions that are not critical scoring errors (CSEs). We conduct the experiments in our new task formulation and demonstrate that a typical SAS system can predict scores with zero CSE for approximately 50% of test data at maximum by filtering out low-reliablility predictions on the basis of a certain confidence estimation. This result directly indicates the possibility of reducing half the scoring cost of human raters, which is more preferable for the evaluation of SAS systems.
Anthology ID:
2020.acl-srw.32
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
Month:
July
Year:
2020
Address:
Online
Editors:
Shruti Rijhwani, Jiangming Liu, Yizhong Wang, Rotem Dror
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
237–243
Language:
URL:
https://aclanthology.org/2020.acl-srw.32
DOI:
10.18653/v1/2020.acl-srw.32
Bibkey:
Cite (ACL):
Hiroaki Funayama, Shota Sasaki, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, Masato Mita, and Kentaro Inui. 2020. Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 237–243, Online. Association for Computational Linguistics.
Cite (Informal):
Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation (Funayama et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-srw.32.pdf
Video:
 http://slideslive.com/38928677