CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations

Rakesh R. Menon, Sayan Ghosh, Shashank Srivastava


Abstract
Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, a hallmark of human intelligence is the ability to learn new concepts purely from language. Here, we explore training zero-shot classifiers for structured data purely from language. For this, we introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classification tasks over structured data along with natural language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. To model the influence of explanations in classifying an example, we develop ExEnt, an entailment-based model that learns classifiers using explanations. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations. We delineate key challenges for automated learning from explanations, addressing which can lead to progress on CLUES in the future. Code and datasets are available at: https://clues-benchmark.github.io.
Anthology ID:
2022.acl-long.451
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6523–6546
Language:
URL:
https://aclanthology.org/2022.acl-long.451
DOI:
10.18653/v1/2022.acl-long.451
Bibkey:
Cite (ACL):
Rakesh R. Menon, Sayan Ghosh, and Shashank Srivastava. 2022. CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6523–6546, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations (R. Menon et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.451.pdf
Software:
 2022.acl-long.451.software.zip
Data
CLUES (Classifier Learning Using natural language ExplanationS)MultiNLI