Revisiting the Evaluation for Cross Document Event Coreference

Shyam Upadhyay, Nitish Gupta, Christos Christodoulopoulos, Dan Roth


Abstract
Cross document event coreference (CDEC) is an important task that aims at aggregating event-related information across multiple documents. We revisit the evaluation for CDEC, and discover that past works have adopted different, often inconsistent, evaluation settings, which either overlook certain mistakes in coreference decisions, or make assumptions that simplify the coreference task considerably. We suggest a new evaluation methodology which overcomes these limitations, and allows for an accurate assessment of CDEC systems. Our new evaluation setting better reflects the corpus-wide information aggregation ability of CDEC systems by separating event-coreference decisions made across documents from those made within a document. In addition, we suggest a better baseline for the task and semi-automatically identify several inconsistent annotations in the evaluation dataset.
Anthology ID:
C16-1183
Volume:
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Month:
December
Year:
2016
Address:
Osaka, Japan
Editors:
Yuji Matsumoto, Rashmi Prasad
Venue:
COLING
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
1949–1958
Language:
URL:
https://aclanthology.org/C16-1183
DOI:
Bibkey:
Cite (ACL):
Shyam Upadhyay, Nitish Gupta, Christos Christodoulopoulos, and Dan Roth. 2016. Revisiting the Evaluation for Cross Document Event Coreference. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1949–1958, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Revisiting the Evaluation for Cross Document Event Coreference (Upadhyay et al., COLING 2016)
Copy Citation:
PDF:
https://aclanthology.org/C16-1183.pdf
Data
ECB+