SpaceNLI: Evaluating the Consistency of Predicting Inferences In Space

Lasha Abzianidze, Joost Zwarts, Yoad Winter


Abstract
While many natural language inference (NLI) datasets target certain semantic phenomena, e.g., negation, tense & aspect, monotonicity, and presupposition, to the best of our knowledge, there is no NLI dataset that involves diverse types of spatial expressions and reasoning. We fill this gap by semi-automatically creating an NLI dataset for spatial reasoning, called SpaceNLI. The data samples are automatically generated from a curated set of reasoning patterns (see Figure 1), where the patterns are annotated with inference labels by experts. We test several SOTA NLI systems on SpaceNLI to gauge the complexity of the dataset and the system’s capacity for spatial reasoning. Moreover, we introduce a Pattern Accuracy and argue that it is a more reliable and stricter measure than the accuracy for evaluating a system’s performance on pattern-based generated data samples. Based on the evaluation results we find that the systems obtain moderate results on the spatial NLI problems but lack consistency per inference pattern. The results also reveal that non-projective spatial inferences (especially due to the “between” preposition) are the most challenging ones.
Anthology ID:
2023.naloma-1.2
Volume:
Proceedings of the 4th Natural Logic Meets Machine Learning Workshop
Month:
June
Year:
2023
Address:
Nancy, France
Editors:
Stergios Chatzikyriakidis, Valeria de Paiva
Venues:
NALOMA | WS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
12–24
Language:
URL:
https://aclanthology.org/2023.naloma-1.2
DOI:
Bibkey:
Cite (ACL):
Lasha Abzianidze, Joost Zwarts, and Yoad Winter. 2023. SpaceNLI: Evaluating the Consistency of Predicting Inferences In Space. In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pages 12–24, Nancy, France. Association for Computational Linguistics.
Cite (Informal):
SpaceNLI: Evaluating the Consistency of Predicting Inferences In Space (Abzianidze et al., NALOMA-WS 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.naloma-1.2.pdf
Optional supplementary material:
 2023.naloma-1.2.OptionalSupplementaryMaterial.zip