Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian, James V. Bruno


Abstract
This study aims to build an automatic system for the detection of plagiarized spoken responses in the context of an assessment of English speaking proficiency for non-native speakers. Classification models were trained to distinguish between plagiarized and non-plagiarized responses with two different types of features: text-to-text content similarity measures, which are commonly used in the task of plagiarism detection for written documents, and speaking proficiency measures, which were specifically designed for spontaneous speech and extracted using an automated speech scoring system. The experiments were first conducted on a large data set drawn from an operational English proficiency assessment across multiple years, and the best classifier on this heavily imbalanced data set resulted in an F1-score of 0.761 on the plagiarized class. This system was then validated on operational responses collected from a single administration of the assessment and achieved a recall of 0.897. The results indicate that the proposed system can potentially be used to improve the validity of both human and automated assessment of non-native spoken English.
Anthology ID:
W19-4445
Volume:
Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
August
Year:
2019
Address:
Florence, Italy
Venues:
ACL | BEA | WS
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
435–443
Language:
URL:
https://aclanthology.org/W19-4445
DOI:
10.18653/v1/W19-4445
Bibkey:
Cite (ACL):
Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian, and James V. Bruno. 2019. Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 435–443, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency (Wang et al., 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-4445.pdf