Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays

Zarah Weiss, Anja Riemenschneider, Pauline Schröter, Detmar Meurers


Abstract
Computational linguistic research on the language complexity of student writing typically involves human ratings as a gold standard. However, educational science shows that teachers find it difficult to identify and cleanly separate accuracy, different aspects of complexity, contents, and structure. In this paper, we therefore explore the use of computational linguistic methods to investigate how task-appropriate complexity and accuracy relate to the grading of overall performance, content performance, and language performance as assigned by teachers. Based on texts written by students for the official school-leaving state examination (Abitur), we show that teachers successfully assign higher language performance grades to essays with higher task-appropriate language complexity and properly separate this from content scores. Yet, accuracy impacts teacher assessment for all grading rubrics, also the content score, overemphasizing the role of accuracy. Our analysis is based on broad computational linguistic modeling of German language complexity and an innovative theory- and data-driven feature aggregation method inferring task-appropriate language complexity.
Anthology ID:
W19-4404
Volume:
Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
August
Year:
2019
Address:
Florence, Italy
Editors:
Helen Yannakoudakis, Ekaterina Kochmar, Claudia Leacock, Nitin Madnani, Ildikó Pilán, Torsten Zesch
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
30–45
Language:
URL:
https://aclanthology.org/W19-4404
DOI:
10.18653/v1/W19-4404
Bibkey:
Cite (ACL):
Zarah Weiss, Anja Riemenschneider, Pauline Schröter, and Detmar Meurers. 2019. Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 30–45, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays (Weiss et al., BEA 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-4404.pdf