Verification of Reasoning Ability using BDI Logic and Large Language Model in AIWolf

Hiraku Gondo, Hiroki Sakaji, Itsuki Noda


Abstract
We attempt to improve the reasoning capability of LLMs in werewolf game by combining BDI logic with LLMs. While LLMs such as ChatGPT has been developed and used for various tasks, there remain several weakness of the LLMs. Logical reasoning is one of such weakness. Therefore, we try to introduce BDI logic-based prompts to verify the logical reasoning ability of LLMs in dialogue of werewofl game. Experiments and evaluations were conducted using “AI-Werewolf,” a communication game for AI with incomplete information. From the results of the game played by five agents, we compare the logical reasoning ability of LLMs by using the win rate and the vote rate against werewolf.
Anthology ID:
2024.aiwolfdial-1.5
Volume:
Proceedings of the 2nd International AIWolfDial Workshop
Month:
September
Year:
2024
Address:
Tokyo, Japan
Editor:
Yoshinobu Kano
Venues:
AIWolfDial | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
40–47
Language:
URL:
https://aclanthology.org/2024.aiwolfdial-1.5
DOI:
Bibkey:
Cite (ACL):
Hiraku Gondo, Hiroki Sakaji, and Itsuki Noda. 2024. Verification of Reasoning Ability using BDI Logic and Large Language Model in AIWolf. In Proceedings of the 2nd International AIWolfDial Workshop, pages 40–47, Tokyo, Japan. Association for Computational Linguistics.
Cite (Informal):
Verification of Reasoning Ability using BDI Logic and Large Language Model in AIWolf (Gondo et al., AIWolfDial-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.aiwolfdial-1.5.pdf
Supplementary attachment:
 2024.aiwolfdial-1.5.Supplementary_Attachment.zip