yangqi at SemEval-2024 Task 9: Simulate Human Thinking by Large Language Model for Lateral Thinking Challenges

Qi Yang, Jingjie Zeng, Liang Yang, Hongfei Lin


Abstract
This paper describes our system used in the SemEval-2024 Task 9 on two sub-tasks, BRAINTEASER: A Novel Task Defying Common Sense. In this work, we developed a system SHTL, which means simulate human thinking capabilities by Large Language Model (LLM). Our approach bifurcates into two main components: Common Sense Reasoning and Rationalize Defying Common Sense. To mitigate the hallucinations of LLM, we implemented a strategy that combines Retrieval-augmented Generation (RAG) with the the Self-Adaptive In-Context Learning (SAICL), thereby sufficiently leveraging the powerful language ability of LLM. The effectiveness of our method has been validated by its performance on the test set, with an average performance on two subtasks that is 30.1 higher than ChatGPT setting zero-shot and only 0.8 lower than that of humans.
Anthology ID:
2024.semeval-1.36
Volume:
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
233–238
Language:
URL:
https://aclanthology.org/2024.semeval-1.36
DOI:
10.18653/v1/2024.semeval-1.36
Bibkey:
Cite (ACL):
Qi Yang, Jingjie Zeng, Liang Yang, and Hongfei Lin. 2024. yangqi at SemEval-2024 Task 9: Simulate Human Thinking by Large Language Model for Lateral Thinking Challenges. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 233–238, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
yangqi at SemEval-2024 Task 9: Simulate Human Thinking by Large Language Model for Lateral Thinking Challenges (Yang et al., SemEval 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.semeval-1.36.pdf
Supplementary material:
 2024.semeval-1.36.SupplementaryMaterial.zip
Supplementary material:
 2024.semeval-1.36.SupplementaryMaterial.txt