Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension

Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, Peter Clark


Abstract
We present a new dataset and models for comprehending paragraphs about processes (e.g., photosynthesis), an important genre of text describing a dynamic world. The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints). The end-task, tracking the location and existence of entities through the text, is challenging because the causal effects of actions are often implicit and need to be inferred. We find that previous models that have worked well on synthetic data achieve only mediocre performance on ProPara, and introduce two new neural models that exploit alternative mechanisms for state prediction, in particular using LSTM input encoding and span prediction. The new models improve accuracy by up to 19%. We are releasing the ProPara dataset and our models to the community.
Anthology ID:
N18-1144
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1595–1604
Language:
URL:
https://aclanthology.org/N18-1144
DOI:
10.18653/v1/N18-1144
Bibkey:
Cite (ACL):
Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1595–1604, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension (Dalvi et al., NAACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/N18-1144.pdf
Data
ProParaSQuAD