Scoring Interactional Aspects of Human-Machine Dialog for Language Learning and Assessment using Text Features

Vikram Ramanarayanan, Matthew Mulholland, Yao Qian


Abstract
While there has been much work in the language learning and assessment literature on human and automated scoring of essays and short constructed responses, there is little to no work examining text features for scoring of dialog data, particularly interactional aspects thereof, to assess conversational proficiency over and above constructed response skills. Our work bridges this gap by investigating both human and automated approaches towards scoring human–machine text dialog in the context of a real-world language learning application. We collected conversational data of human learners interacting with a cloud-based standards-compliant dialog system, triple-scored these data along multiple dimensions of conversational proficiency, and then analyzed the performance trends. We further examined two different approaches to automated scoring of such data and show that these approaches are able to perform at or above par with human agreement for a majority of dimensions of the scoring rubric.
Anthology ID:
W19-5913
Volume:
Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue
Month:
September
Year:
2019
Address:
Stockholm, Sweden
Venues:
SIGDIAL | WS
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
103–109
Language:
URL:
https://aclanthology.org/W19-5913
DOI:
10.18653/v1/W19-5913
Bibkey:
Cite (ACL):
Vikram Ramanarayanan, Matthew Mulholland, and Yao Qian. 2019. Scoring Interactional Aspects of Human-Machine Dialog for Language Learning and Assessment using Text Features. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 103–109, Stockholm, Sweden. Association for Computational Linguistics.
Cite (Informal):
Scoring Interactional Aspects of Human-Machine Dialog for Language Learning and Assessment using Text Features (Ramanarayanan et al., 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-5913.pdf