Linguistic, Kinematic and Gaze Information in Task Descriptions: The LKG-Corpus

Tim Reinboth, Stephanie Gross, Laura Bishop, Brigitte Krenn


Abstract
Data from neuroscience and psychology suggest that sensorimotor cognition may be of central importance to language. Specifically, the linguistic structure of utterances referring to concrete actions may reflect the structure of the sensorimotor processing underlying the same action. To investigate this, we present the Linguistic, Kinematic and Gaze information in task descriptions Corpus (LKG-Corpus), comprising multimodal data on 13 humans, conducting take, put, and push actions, and describing these actions with 350 utterances. Recorded are audio, video, motion and eye-tracking data while participants perform an action and describe what they do. The dataset is annotated with orthographic transcriptions of utterances and information on: (a) gaze behaviours, (b) when a participant touched an object, (c) when an object was moved, (d) when a participant looked at the location s/he would next move the object to, (e) when the participant’s gaze was stable on an area. With the exception of the annotation of stable gaze, all annotations were performed manually. With the LKG-Corpus, we present a dataset that integrates linguistic, kinematic and gaze data with an explicit focus on relations between action and language. On this basis, we outline applications of the dataset to both basic and applied research.
Anthology ID:
2020.lrec-1.19
Volume:
Proceedings of the 12th Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
147–155
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.19
DOI:
Bibkey:
Cite (ACL):
Tim Reinboth, Stephanie Gross, Laura Bishop, and Brigitte Krenn. 2020. Linguistic, Kinematic and Gaze Information in Task Descriptions: The LKG-Corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 147–155, Marseille, France. European Language Resources Association.
Cite (Informal):
Linguistic, Kinematic and Gaze Information in Task Descriptions: The LKG-Corpus (Reinboth et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.19.pdf