Open-ended Knowledge Tracing for Computer Science Education

Naiming Liu, Zichao Wang, Richard Baraniuk, Andrew Lan


Abstract
In educational applications, knowledge tracing refers to the problem of estimating students’ time-varying concept/skill mastery level from their past responses to questions and predicting their future performance. One key limitation of most existing knowledge tracing methods is that they treat student responses to questions as binary-valued, i.e., whether they are correct or incorrect. Response correctness analysis/prediction is straightforward, but it ignores important information regarding mastery, especially for open-ended questions. In contrast, exact student responses can provide much more information. In this paper, we conduct the first exploration int open-ended knowledge tracing (OKT) by studying the new task of predicting students’ exact open-ended responses to questions. Our work is grounded in the domain of computer science education with programming questions. We develop an initial solution to the OKT problem, a student knowledge-guided code generation approach, that combines program synthesis methods using language models with student knowledge tracing methods. We also conduct a series of quantitative and qualitative experiments on a real-world student code dataset to validate and demonstrate the promise of OKT.
Anthology ID:
2022.emnlp-main.254
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3849–3862
Language:
URL:
https://aclanthology.org/2022.emnlp-main.254
DOI:
10.18653/v1/2022.emnlp-main.254
Bibkey:
Cite (ACL):
Naiming Liu, Zichao Wang, Richard Baraniuk, and Andrew Lan. 2022. Open-ended Knowledge Tracing for Computer Science Education. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3849–3862, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Open-ended Knowledge Tracing for Computer Science Education (Liu et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.254.pdf