ACTA: Short-Answer Grading in High-Stakes Medical Exams

King Yiu Suen, Victoria Yaneva, Le An Ha, Janet Mee, Yiyun Zhou, Polina Harik


Abstract
This paper presents the ACTA system, which performs automated short-answer grading in the domain of high-stakes medical exams. The system builds upon previous work on neural similarity-based grading approaches by applying these to the medical domain and utilizing contrastive learning as a means to optimize the similarity metric. ACTA is evaluated against three strong baselines and is developed in alignment with operational needs, where low-confidence responses are flagged for human review. Learning curves are explored to understand the effects of training data on performance. The results demonstrate that ACTA leads to substantially lower number of responses being flagged for human review, while maintaining high classification accuracy.
Anthology ID:
2023.bea-1.36
Volume:
Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaïs Tack, Victoria Yaneva, Zheng Yuan, Torsten Zesch
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
443–447
Language:
URL:
https://aclanthology.org/2023.bea-1.36
DOI:
10.18653/v1/2023.bea-1.36
Bibkey:
Cite (ACL):
King Yiu Suen, Victoria Yaneva, Le An Ha, Janet Mee, Yiyun Zhou, and Polina Harik. 2023. ACTA: Short-Answer Grading in High-Stakes Medical Exams. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 443–447, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
ACTA: Short-Answer Grading in High-Stakes Medical Exams (Suen et al., BEA 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.bea-1.36.pdf