Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models

Tassilo Klein, Moin Nabi


Abstract
Can we get existing language models and refine them for zero-shot commonsense reasoning? This paper presents an initial study exploring the feasibility of zero-shot commonsense reasoning for the Winograd Schema Challenge by formulating the task as self-supervised refinement of a pre-trained language model. In contrast to previous studies that rely on fine-tuning annotated datasets, we seek to boost conceptualization via loss landscape refinement. To this end, we propose a novel self-supervised learning approach that refines the language model utilizing a set of linguistic perturbations of similar concept relationships. Empirical analysis of our conceptually simple framework demonstrates the viability of zero-shot commonsense reasoning on multiple benchmarks.
Anthology ID:
2021.emnlp-main.688
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8737–8743
Language:
URL:
https://aclanthology.org/2021.emnlp-main.688
DOI:
10.18653/v1/2021.emnlp-main.688
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.688.pdf
Code
 sap-samples/emnlp2021-contrastive-refinement
Data
GAP Coreference DatasetWSCWinoBiasWinoGrande