Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition

Elena V. Epure, Romain Hennequin


Abstract
Multiple works have proposed to probe language models (LMs) for generalization in named entity (NE) typing (NET) and recognition (NER). However, little has been done in this direction for auto-regressive models despite their popularity and potential to express a wide variety of NLP tasks in the same unified format. We propose a new methodology to probe auto-regressive LMs for NET and NER generalization, which draws inspiration from human linguistic behavior, by resorting to meta-learning. We study NEs of various types individually by designing a zero-shot transfer strategy for NET. Then, we probe the model for NER by providing a few examples at inference. We introduce a novel procedure to assess the model’s memorization of NEs and report the memorization’s impact on the results. Our findings show that: 1) GPT2, a common pre-trained auto-regressive LM, without any fine-tuning for NET or NER, performs the tasksfairly well; 2) name irregularity when common for a NE type could be an effective exploitable cue; 3) the model seems to rely more on NE than contextual cues in few-shot NER; 4) NEs with words absent during LM pre-training are very challenging for both NET and NER.
Anthology ID:
2022.lrec-1.151
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
1408–1417
Language:
URL:
https://aclanthology.org/2022.lrec-1.151
DOI:
Bibkey:
Cite (ACL):
Elena V. Epure and Romain Hennequin. 2022. Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1408–1417, Marseille, France. European Language Resources Association.
Cite (Informal):
Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition (Epure & Hennequin, LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.151.pdf
Code
 deezer/net-ner-probing
Data
DBpediaWNUT 2017