DREAM: Improving Situational QA by First Elaborating the Situation

Yuling Gu, Bhavana Dalvi, Peter Clark


Abstract
When people answer questions about a specific situation, e.g., “I cheated on my mid-term exam last week. Was that wrong?”, cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the “scene”. To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further fine-tuning the QA system on DREAM’s training data. These results suggest that adding focused elaborations about a situation can improve a system’s reasoning about it, and may serve as an effective way of injecting new scenario-based knowledge into QA models. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters.
Anthology ID:
2022.naacl-main.82
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1115–1127
Language:
URL:
https://aclanthology.org/2022.naacl-main.82
DOI:
10.18653/v1/2022.naacl-main.82
Bibkey:
Cite (ACL):
Yuling Gu, Bhavana Dalvi, and Peter Clark. 2022. DREAM: Improving Situational QA by First Elaborating the Situation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1127, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
DREAM: Improving Situational QA by First Elaborating the Situation (Gu et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.82.pdf
Video:
 https://aclanthology.org/2022.naacl-main.82.mp4
Code
 allenai/dream
Data
CODAHETHICSMoral StoriesStory Commonsense