GeminiPro at SemEval-2024 Task 9: BrainTeaser on Gemini

Kyu Hyun Choi, Seung-hoon Na


Abstract
It is known that human thought can be distinguished into lateral and vertical thinking. The development of language models has thus far been focused on evaluating and advancing vertical thinking, while lateral thinking has been somewhat neglected. To foster progress in this area, SemEval has created and distributed a brainteaser dataset based on lateral thinking consist of sentence puzzles and word puzzle QA. In this paper, we test and discuss the performance of the currently known best model, Gemini, on this dataset.
Anthology ID:
2024.semeval-1.228
Volume:
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
1602–1606
Language:
URL:
https://aclanthology.org/2024.semeval-1.228
DOI:
10.18653/v1/2024.semeval-1.228
Bibkey:
Cite (ACL):
Kyu Hyun Choi and Seung-hoon Na. 2024. GeminiPro at SemEval-2024 Task 9: BrainTeaser on Gemini. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1602–1606, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
GeminiPro at SemEval-2024 Task 9: BrainTeaser on Gemini (Choi & Na, SemEval 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.semeval-1.228.pdf
Supplementary material:
 2024.semeval-1.228.SupplementaryMaterial.txt
Supplementary material:
 2024.semeval-1.228.SupplementaryMaterial.zip