‘Meet me at the ribary’ – Acceptability of spelling variants in free-text answers to listening comprehension prompts

Ronja Laarmann-Quante, Leska Schwarz, Andrea Horbach, Torsten Zesch


Abstract
When listening comprehension is tested as a free-text production task, a challenge for scoring the answers is the resulting wide range of spelling variants. When judging whether a variant is acceptable or not, human raters perform a complex holistic decision. In this paper, we present a corpus study in which we analyze human acceptability decisions in a high stakes test for German. We show that for human experts, spelling variants are harder to score consistently than other answer variants.Furthermore, we examine how the decision can be operationalized using features that could be applied by an automatic scoring system. We show that simple measures like edit distance and phonetic similarity between a given answer and the target answer can model the human acceptability decisions with the same inter-annotator agreement as humans, and discuss implications of the remaining inconsistencies.
Anthology ID:
2022.bea-1.22
Volume:
Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)
Month:
July
Year:
2022
Address:
Seattle, Washington
Venues:
BEA | NAACL
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
173–182
Language:
URL:
https://aclanthology.org/2022.bea-1.22
DOI:
10.18653/v1/2022.bea-1.22
Bibkey:
Cite (ACL):
Ronja Laarmann-Quante, Leska Schwarz, Andrea Horbach, and Torsten Zesch. 2022. ‘Meet me at the ribary’ – Acceptability of spelling variants in free-text answers to listening comprehension prompts. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 173–182, Seattle, Washington. Association for Computational Linguistics.
Cite (Informal):
‘Meet me at the ribary’ – Acceptability of spelling variants in free-text answers to listening comprehension prompts (Laarmann-Quante et al., BEA 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.bea-1.22.pdf