BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models

Jacek Wiland, Max Ploner, Alan Akbik


Abstract
Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurations. However, previous approaches rely on the objective function used in pre-training LMs and are thus applicable only to masked or causal LMs. As a result, comparing different types of LMs becomes impossible. To address this, we propose an approach that uses an LM’s inherent ability to estimate the log-likelihood of any given textual statement. We carefully design an evaluation dataset of 7,731 instances (40,916 in a larger variant) from which we produce alternative statements for each relational fact, one of which is correct. We then evaluate whether an LM correctly assigns the highest log-likelihood to the correct statement. Our experimental evaluation of 22 common LMs shows that our proposed framework, BEAR, can effectively probe for knowledge across different LM types. We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs.
Anthology ID:
2024.findings-naacl.155
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2393–2411
Language:
URL:
https://aclanthology.org/2024.findings-naacl.155
DOI:
Bibkey:
Cite (ACL):
Jacek Wiland, Max Ploner, and Alan Akbik. 2024. BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2393–2411, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models (Wiland et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.155.pdf
Copyright:
 2024.findings-naacl.155.copyright.pdf