Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment

Nanjiang Jiang, Marie-Catherine de Marneffe


Abstract
When a speaker, Mary, asks “Do you know that Florence is packed with visitors?”, we take her to believe that Florence is packed with visitors, but not if she asks “Do you think that Florence is packed with visitors?”. Inferring speaker commitment (aka event factuality) is crucial for information extraction and question answering. Here, we explore the hypothesis that linguistic deficits drive the error patterns of existing speaker commitment models by analyzing the linguistic correlates of model error on a challenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement (“Florence is packed with visitors” in our example) of clause-embedding verbs (“know”, “think”) under four entailment-canceling environments (negation, modal, question, conditional). A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic constructions (e.g., conditionals) in natural language, highlighting directions for improvement.
Anthology ID:
P19-1412
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4208–4213
Language:
URL:
https://aclanthology.org/P19-1412
DOI:
10.18653/v1/P19-1412
Award:
 Best Short Paper
Bibkey:
Cite (ACL):
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4208–4213, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment (Jiang & de Marneffe, ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1412.pdf
Video:
 https://vimeo.com/385255521