Cheater’s Bowl: Human vs. Computer Search Strategies for Open-Domain QA

Wanrong He, Andrew Mao, Jordan Boyd-Graber


Abstract
For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus. However, the strategies that computers use fundamentally differ from those of humans. To better understand these differences, we design a gamified interface for data collection—Cheater’s Bowl—where a human answers complex questions with access to both traditional and modern search tools. We collect a dataset of human search sessions, analyze human search strategies, and compare them to state-of-the-art multi-hop QA models. Humans query logically, apply dynamic search chains, and use world knowledge to boost searching. We demonstrate how human queries can improve the accuracy of existing systems and propose improving the future design of QA models.
Anthology ID:
2022.findings-emnlp.266
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2022
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3627–3639
Language:
URL:
https://aclanthology.org/2022.findings-emnlp.266
DOI:
10.18653/v1/2022.findings-emnlp.266
Bibkey:
Cite (ACL):
Wanrong He, Andrew Mao, and Jordan Boyd-Graber. 2022. Cheater’s Bowl: Human vs. Computer Search Strategies for Open-Domain QA. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3627–3639, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Cheater’s Bowl: Human vs. Computer Search Strategies for Open-Domain QA (He et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-emnlp.266.pdf
Video:
 https://aclanthology.org/2022.findings-emnlp.266.mp4