SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, Parisa Kordjamshidi


Abstract
This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs’ capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.
Anthology ID:
2021.naacl-main.364
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4582–4598
Language:
URL:
https://aclanthology.org/2021.naacl-main.364
DOI:
10.18653/v1/2021.naacl-main.364
Bibkey:
Cite (ACL):
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582–4598, Online. Association for Computational Linguistics.
Cite (Informal):
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning (Mirzaee et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.364.pdf
Video:
 https://aclanthology.org/2021.naacl-main.364.mp4
Code
 HLR/SpartQA-baselines +  additional community code
Data
SPARTQA -BoolQGLUENLVR