NIMZ at SemEval-2024 Task 9: Evaluating Methods in Solving Brainteasers Defying Commonsense

Zahra Rahimi, Mohammad Moein Shirzady, Zeinab Taghavi, Hossein Sameti


Abstract
The goal and dream of the artificial intelligence field have long been the development of intelligent systems or agents that mimic human behavior and thinking. Creativity is an essential trait in humans that is closely related to lateral thinking. The remarkable advancements in Language Models have led to extensive research on question-answering and explicit and implicit reasoning involving vertical thinking. However, there is an increasing need to shift focus towards research and development of models that can think laterally. One must step outside the traditional frame of commonsense concepts in lateral thinking to conclude. Task 9 of SemEval-2024 is Brainteaser (Jiang et al.,2024), which requires lateral thinking to answer riddle-like multiple-choice questions. In our study, we assessed the performance of various models for the Brainteaser task. We achieved an overall accuracy of 75% for the Sentence Puzzle subtask and 66.7% for the Word Puzzle subtask. All the codes, along with the links to our saved models, are available on our GitHub.
Anthology ID:
2024.semeval-1.23
Volume:
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
148–154
Language:
URL:
https://aclanthology.org/2024.semeval-1.23
DOI:
10.18653/v1/2024.semeval-1.23
Bibkey:
Cite (ACL):
Zahra Rahimi, Mohammad Moein Shirzady, Zeinab Taghavi, and Hossein Sameti. 2024. NIMZ at SemEval-2024 Task 9: Evaluating Methods in Solving Brainteasers Defying Commonsense. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 148–154, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
NIMZ at SemEval-2024 Task 9: Evaluating Methods in Solving Brainteasers Defying Commonsense (Rahimi et al., SemEval 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.semeval-1.23.pdf
Supplementary material:
 2024.semeval-1.23.SupplementaryMaterial.zip
Supplementary material:
 2024.semeval-1.23.SupplementaryMaterial.txt