Static Embeddings as Efficient Knowledge Bases?

Philipp Dufter, Nora Kassner, Hinrich Schütze


Abstract
Recent research investigates factual knowledge stored in large pretrained language models (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. The good performance on this analysis task has been interpreted as PLMs becoming potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform 1.6% points better than BERT while just using 0.3% of energy for training. One important factor in their good comparative performance is that static embeddings are standardly learned for a large vocabulary. In contrast, BERT exploits its more sophisticated, but expensive ability to compose meaningful representations from a much smaller subword vocabulary.
Anthology ID:
2021.naacl-main.186
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2353–2363
Language:
URL:
https://aclanthology.org/2021.naacl-main.186
DOI:
10.18653/v1/2021.naacl-main.186
Bibkey:
Cite (ACL):
Philipp Dufter, Nora Kassner, and Hinrich Schütze. 2021. Static Embeddings as Efficient Knowledge Bases?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2353–2363, Online. Association for Computational Linguistics.
Cite (Informal):
Static Embeddings as Efficient Knowledge Bases? (Dufter et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.186.pdf
Video:
 https://aclanthology.org/2021.naacl-main.186.mp4
Code
 pdufter/staticlama
Data
LAMAT-REx