Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

Avia Efrat, Uri Shaham, Dan Kilman, Omer Levy


Abstract
Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, whose solving requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for experienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryptonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%).
Anthology ID:
2021.emnlp-main.344
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4186–4192
Language:
URL:
https://aclanthology.org/2021.emnlp-main.344
DOI:
10.18653/v1/2021.emnlp-main.344
Bibkey:
Cite (ACL):
Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy. 2021. Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4186–4192, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language (Efrat et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.344.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.344.mp4
Code
 aviaefrat/cryptonite