Analytic Automated Essay Scoring Based on Deep Neural Networks Integrating Multidimensional Item Response Theory

Takumi Shibata, Masaki Uto


Abstract
Essay exams have been attracting attention as a way of measuring the higher-order abilities of examinees, but they have two major drawbacks in that grading them is expensive and raises questions about fairness. As an approach to overcome these problems, automated essay scoring (AES) is in increasing need. Many AES models based on deep neural networks have been proposed in recent years and have achieved high accuracy, but most of these models are designed to predict only a single overall score. However, to provide detailed feedback in practical situations, we often require not only the overall score but also analytic scores corresponding to various aspects of the essay.Several neural AES models that can predict both the analytic scores and the overall score have also been proposed for this very purpose. However, conventional models are designed to have complex neural architectures for each analytic score, which makes interpreting the score prediction difficult. To improve the interpretability of the prediction while maintaining scoring accuracy, we propose a new neural model for automated analytic scoring that integrates a multidimensional item response theory model, which is a popular psychometric model.
Anthology ID:
2022.coling-1.257
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
2917–2926
Language:
URL:
https://aclanthology.org/2022.coling-1.257
DOI:
Bibkey:
Cite (ACL):
Takumi Shibata and Masaki Uto. 2022. Analytic Automated Essay Scoring Based on Deep Neural Networks Integrating Multidimensional Item Response Theory. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2917–2926, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Analytic Automated Essay Scoring Based on Deep Neural Networks Integrating Multidimensional Item Response Theory (Shibata & Uto, COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.257.pdf