MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling

Parishad BehnamGhader, Hossein Zakerinia, Mahdieh Soleymani Baghshah


Abstract
Pre-trained models like Bidirectional Encoder Representations from Transformers (BERT), have recently made a big leap forward in Natural Language Processing (NLP) tasks. However, there are still some shortcomings in the Masked Language Modeling (MLM) task performed by these models. In this paper, we first introduce a multi-graph including different types of relations between words. Then, we propose Multi-Graph augmented BERT (MG-BERT) model that is based on BERT. MG-BERT embeds tokens while taking advantage of a static multi-graph containing global word co-occurrences in the text corpus beside global real-world facts about words in knowledge graphs. The proposed model also employs a dynamic sentence graph to capture local context effectively. Experimental results demonstrate that our model can considerably enhance the performance in the MLM task.
Anthology ID:
2021.textgraphs-1.12
Volume:
Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)
Month:
June
Year:
2021
Address:
Mexico City, Mexico
Editors:
Alexander Panchenko, Fragkiskos D. Malliaros, Varvara Logacheva, Abhik Jana, Dmitry Ustalov, Peter Jansen
Venue:
TextGraphs
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
125–131
Language:
URL:
https://aclanthology.org/2021.textgraphs-1.12
DOI:
10.18653/v1/2021.textgraphs-1.12
Bibkey:
Cite (ACL):
Parishad BehnamGhader, Hossein Zakerinia, and Mahdieh Soleymani Baghshah. 2021. MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15), pages 125–131, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling (BehnamGhader et al., TextGraphs 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.textgraphs-1.12.pdf
Optional supplementary material:
 2021.textgraphs-1.12.OptionalSupplementaryMaterial.zip
Data
CoLASSTSST-2