Pre-training Entity Relation Encoder with Intra-span and Inter-span Information

Yijun Wang, Changzhi Sun, Yuanbin Wu, Junchi Yan, Peng Gao, Guotong Xie


Abstract
In this paper, we integrate span-related information into pre-trained encoder for entity relation extraction task. Instead of using general-purpose sentence encoder (e.g., existing universal pre-trained models), we introduce a span encoder and a span pair encoder to the pre-training network, which makes it easier to import intra-span and inter-span information into the pre-trained model. To learn the encoders, we devise three customized pre-training objectives from different perspectives, which target on tokens, spans, and span pairs. In particular, a span encoder is trained to recover a random shuffling of tokens in a span, and a span pair encoder is trained to predict positive pairs that are from the same sentences and negative pairs that are from different sentences using contrastive loss. Experimental results show that the proposed pre-training method outperforms distantly supervised pre-training, and achieves promising performance on two entity relation extraction benchmark datasets (ACE05, SciERC).
Anthology ID:
2020.emnlp-main.132
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1692–1705
Language:
URL:
https://aclanthology.org/2020.emnlp-main.132
DOI:
10.18653/v1/2020.emnlp-main.132
Bibkey:
Cite (ACL):
Yijun Wang, Changzhi Sun, Yuanbin Wu, Junchi Yan, Peng Gao, and Guotong Xie. 2020. Pre-training Entity Relation Encoder with Intra-span and Inter-span Information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1692–1705, Online. Association for Computational Linguistics.
Cite (Informal):
Pre-training Entity Relation Encoder with Intra-span and Inter-span Information (Wang et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.132.pdf
Video:
 https://slideslive.com/38939251