ECWCA - Educational CrossWord Clues Answering: A CALAMITA Challenge

Andrea Zugarini, Kamyar Zeinalipour, Achille Fusco, Asya Zanollo


Abstract
This paper presents ECWCA (Educational CrossWord Clues Answering), a novel challenge designed to evaluate knowledge and reasoning capabilities of large language models through crossword clue-answering. The challenge consists of two tasks: a standard question-answering format where the LLM has to solve crossword clues, and a variation of it, where the model is receives hints about the word lengths of the answers, which is expected to help models with reasoning abilities. To construct the ECWCA dataset, synthetic clues were generated based on entities and facts extracted from Italian Wikipedia. Generated clues were then selected manually in order to ensure high-quality examples with factually correct and unambiguous clues.
Anthology ID:
2024.clicit-1.138
Volume:
Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)
Month:
December
Year:
2024
Address:
Pisa, Italy
Editors:
Felice Dell'Orletta, Alessandro Lenci, Simonetta Montemagni, Rachele Sprugnoli
Venue:
CLiC-it
SIG:
Publisher:
CEUR Workshop Proceedings
Note:
Pages:
1239–1244
Language:
URL:
https://aclanthology.org/2024.clicit-1.138/
DOI:
Bibkey:
Cite (ACL):
Andrea Zugarini, Kamyar Zeinalipour, Achille Fusco, and Asya Zanollo. 2024. ECWCA - Educational CrossWord Clues Answering: A CALAMITA Challenge. In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), pages 1239–1244, Pisa, Italy. CEUR Workshop Proceedings.
Cite (Informal):
ECWCA - Educational CrossWord Clues Answering: A CALAMITA Challenge (Zugarini et al., CLiC-it 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.clicit-1.138.pdf