ChatGPT’s Information Seeking Strategy: Insights from the 20-Questions Game

Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, Raffaella Bernardi


Abstract
Large Language Models, and ChatGPT in particular, have recently grabbed the attention of the community and the media. Having reached high language proficiency, attention has been shifting toward its reasoning capabilities. In this paper, our main aim is to evaluate ChatGPT’s question generation in a task where language production should be driven by an implicit reasoning process. To this end, we employ the 20-Questions game, traditionally used within the Cognitive Science community to inspect the information seeking-strategy’s development. This task requires a series of interconnected skills: asking informative questions, stepwise updating the hypothesis space, and stopping asking questions when enough information has been collected. We build hierarchical hypothesis spaces, exploiting feature norms collected from humans vs. ChatGPT itself, and we inspect the efficiency and informativeness of ChatGPT’s strategy. Our results show that ChatGPT’s performance gets closer to an optimal agent only when prompted to explicitly list the updated space stepwise.
Anthology ID:
2023.inlg-main.11
Volume:
Proceedings of the 16th International Natural Language Generation Conference
Month:
September
Year:
2023
Address:
Prague, Czechia
Editors:
C. Maria Keet, Hung-Yi Lee, Sina Zarrieß
Venues:
INLG | SIGDIAL
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
153–162
Language:
URL:
https://aclanthology.org/2023.inlg-main.11
DOI:
10.18653/v1/2023.inlg-main.11
Bibkey:
Cite (ACL):
Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. 2023. ChatGPT’s Information Seeking Strategy: Insights from the 20-Questions Game. In Proceedings of the 16th International Natural Language Generation Conference, pages 153–162, Prague, Czechia. Association for Computational Linguistics.
Cite (Informal):
ChatGPT’s Information Seeking Strategy: Insights from the 20-Questions Game (Bertolazzi et al., INLG-SIGDIAL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.inlg-main.11.pdf
Supplementary attachment:
 2023.inlg-main.11.Supplementary_Attachment.pdf