Annotating picture description task responses for content analysis

Levi King, Markus Dickinson


Abstract
Given that all users of a language can be creative in their language usage, the overarching goal of this work is to investigate issues of variability and acceptability in written text, for both non-native speakers (NNSs) and native speakers (NSs). We control for meaning by collecting a dataset of picture description task (PDT) responses from a number of NSs and NNSs, and we define and annotate a handful of features pertaining to form and meaning, to capture the multi-dimensional ways in which responses can vary and can be acceptable. By examining the decisions made in this corpus development, we highlight the questions facing anyone working with learner language properties like variability, acceptability and native-likeness. We find reliable inter-annotator agreement, though disagreements point to difficult areas for establishing a link between form and meaning.
Anthology ID:
W18-0510
Volume:
Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Joel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, Helen Yannakoudakis
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
101–109
Language:
URL:
https://aclanthology.org/W18-0510
DOI:
10.18653/v1/W18-0510
Bibkey:
Cite (ACL):
Levi King and Markus Dickinson. 2018. Annotating picture description task responses for content analysis. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 101–109, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Annotating picture description task responses for content analysis (King & Dickinson, BEA 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-0510.pdf
Code
 sailscorpus/sails