Structure-aware Sentence Encoder in Bert-Based Siamese Network

Qiwei Peng, David Weir, Julie Weeds


Abstract
Recently, impressive performance on various natural language understanding tasks has been achieved by explicitly incorporating syntax and semantic information into pre-trained models, such as BERT and RoBERTa. However, this approach depends on problem-specific fine-tuning, and as widely noted, BERT-like models exhibit weak performance, and are inefficient, when applied to unsupervised similarity comparison tasks. Sentence-BERT (SBERT) has been proposed as a general-purpose sentence embedding method, suited to both similarity comparison and downstream tasks. In this work, we show that by incorporating structural information into SBERT, the resulting model outperforms SBERT and previous general sentence encoders on unsupervised semantic textual similarity (STS) datasets and transfer classification tasks.
Anthology ID:
2021.repl4nlp-1.7
Volume:
Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)
Month:
August
Year:
2021
Address:
Online
Editors:
Anna Rogers, Iacer Calixto, Ivan Vulić, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, Vered Shwartz
Venue:
RepL4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
57–63
Language:
URL:
https://aclanthology.org/2021.repl4nlp-1.7
DOI:
10.18653/v1/2021.repl4nlp-1.7
Bibkey:
Cite (ACL):
Qiwei Peng, David Weir, and Julie Weeds. 2021. Structure-aware Sentence Encoder in Bert-Based Siamese Network. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 57–63, Online. Association for Computational Linguistics.
Cite (Informal):
Structure-aware Sentence Encoder in Bert-Based Siamese Network (Peng et al., RepL4NLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.repl4nlp-1.7.pdf
Data
MPQA Opinion CorpusSICKSSTSST-2SST-5SentEval