Behind the Scenes of an Evolving Event Cloze Test

Nathanael Chambers


Abstract
This paper analyzes the narrative event cloze test and its recent evolution. The test removes one event from a document’s chain of events, and systems predict the missing event. Originally proposed to evaluate learned knowledge of event scenarios (e.g., scripts and frames), most recent work now builds ngram-like language models (LM) to beat the test. This paper argues that the test has slowly/unknowingly been altered to accommodate LMs.5 Most notably, tests are auto-generated rather than by hand, and no effort is taken to include core script events. Recent work is not clear on evaluation goals and contains contradictory results. We implement several models, and show that the test’s bias to high-frequency events explains the inconsistencies. We conclude with recommendations on how to return to the test’s original intent, and offer brief suggestions on a path forward.
Anthology ID:
W17-0905
Volume:
Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Michael Roth, Nasrin Mostafazadeh, Nathanael Chambers, Annie Louis
Venue:
LSDSem
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
41–45
Language:
URL:
https://aclanthology.org/W17-0905
DOI:
10.18653/v1/W17-0905
Bibkey:
Cite (ACL):
Nathanael Chambers. 2017. Behind the Scenes of an Evolving Event Cloze Test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 41–45, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Behind the Scenes of an Evolving Event Cloze Test (Chambers, LSDSem 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-0905.pdf