How coherent are neural models of coherence?

Leila Pishdad, Federico Fancellu, Ran Zhang, Afsaneh Fazly


Abstract
Despite the recent advances in coherence modelling, most such models including state-of-the-art neural ones, are evaluated on either contrived proxy tasks such as the standard order discrimination benchmark, or tasks that require special expert annotation. Moreover, most evaluations are conducted on small newswire corpora. To address these shortcomings, in this paper we propose four generic evaluation tasks that draw on different aspects of coherence at both the lexical and document levels, and can be applied to any corpora. In designing these tasks, we aim at capturing coherence-specific properties, such as the correct use of discourse connectives, lexical cohesion, as well as the overall temporal and causal consistency among events and participants in a story. Importantly, our proposed tasks either rely on automatically-generated data, or data annotated for other purposes, hence alleviating the need for annotation specifically targeted to the task of coherence modelling. We perform experiments with several existing state-of-the-art neural models of coherence on these tasks, across large corpora from different domains, including newswire, dialogue, as well as narrative and instructional text. Our findings point to a strong need for revisiting the common practices in the development and evaluation of coherence models.
Anthology ID:
2020.coling-main.539
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6126–6138
Language:
URL:
https://aclanthology.org/2020.coling-main.539
DOI:
10.18653/v1/2020.coling-main.539
Bibkey:
Cite (ACL):
Leila Pishdad, Federico Fancellu, Ran Zhang, and Afsaneh Fazly. 2020. How coherent are neural models of coherence?. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6126–6138, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
How coherent are neural models of coherence? (Pishdad et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.539.pdf
Data
ROCStories