RuleBERT: Teaching Soft Rules to Pre-Trained Language Models

Mohammed Saeed, Naser Ahmadi, Preslav Nakov, Paolo Papotti


Abstract
While pre-trained language models (PLMs) are the go-to solution to tackle many natural language processing problems, they are still very limited in their ability to capture and to use common-sense knowledge. In fact, even if information is available in the form of approximate (soft) logical rules, it is not clear how to transfer it to a PLM in order to improve its performance for deductive reasoning tasks. Here, we aim to bridge this gap by teaching PLMs how to reason with soft Horn rules. We introduce a classification task where, given facts and soft rules, the PLM should return a prediction with a probability for a given hypothesis. We release the first dataset for this task, and we propose a revised loss function that enables the PLM to learn how to predict precise probabilities for the task. Our evaluation results show that the resulting fine-tuned models achieve very high performance, even on logical rules that were unseen at training. Moreover, we demonstrate that logical notions expressed by the rules are transferred to the fine-tuned model, yielding state-of-the-art results on external datasets.
Anthology ID:
2021.emnlp-main.110
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1460–1476
Language:
URL:
https://aclanthology.org/2021.emnlp-main.110
DOI:
10.18653/v1/2021.emnlp-main.110
Bibkey:
Cite (ACL):
Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. 2021. RuleBERT: Teaching Soft Rules to Pre-Trained Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1460–1476, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (Saeed et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.110.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.110.mp4
Code
 mhmdsaiid/rulebert
Data
GLUELAMA