ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations

Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, Nanyun Peng


Abstract
Understanding how events are semantically related to each other is the essence of reading comprehension. Recent event-centric reading comprehension datasets focus mostly on event arguments or temporal relations. While these tasks partially evaluate machines’ ability of narrative understanding, human-like reading comprehension requires the capability to process event-based information beyond arguments and temporal reasoning. For example, to understand causality between events, we need to infer motivation or purpose; to establish event hierarchy, we need to understand the composition of events. To facilitate these tasks, we introduce **ESTER**, a comprehensive machine reading comprehension (MRC) dataset for Event Semantic Relation Reasoning. The dataset leverages natural language queries to reason about the five most common event semantic relations, provides more than 6K questions, and captures 10.1K event relation pairs. Experimental results show that the current SOTA systems achieve 22.1%, 63.3% and 83.5% for token-based exact-match (**EM**), **F1** and event-based **HIT@1** scores, which are all significantly below human performances (36.0%, 79.6%, 100% respectively), highlighting our dataset as a challenging benchmark.
Anthology ID:
2021.emnlp-main.597
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7543–7559
Language:
URL:
https://aclanthology.org/2021.emnlp-main.597
DOI:
10.18653/v1/2021.emnlp-main.597
Bibkey:
Cite (ACL):
Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, and Nanyun Peng. 2021. ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7543–7559, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations (Han et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.597.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.597.mp4
Data
Torque