Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation

Goran Glavaš, Ivan Vulić


Abstract
Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). The recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, however, questions this belief. In this work, we empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks. Relying on the established fine-tuning paradigm, we first couple a pretrained transformer with a biaffine parsing head, aiming to infuse explicit syntactic knowledge from Universal Dependencies treebanks into the transformer. We then fine-tune the model for LU tasks and measure the effect of the intermediate parsing training (IPT) on downstream LU task performance. Results from both monolingual English and zero-shot language transfer experiments (with intermediate target-language parsing) show that explicit formalized syntax, injected into transformers through IPT, has very limited and inconsistent effect on downstream LU performance. Our results, coupled with our analysis of transformers’ representation spaces before and after intermediate parsing, make a significant step towards providing answers to an essential question: how (un)availing is supervised parsing for high-level semantic natural language understanding in the era of large neural models?
Anthology ID:
2021.eacl-main.270
Original:
2021.eacl-main.270v1
Version 2:
2021.eacl-main.270v2
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3090–3104
Language:
URL:
https://aclanthology.org/2021.eacl-main.270
DOI:
10.18653/v1/2021.eacl-main.270
Award:
 Best Long Paper
Bibkey:
Cite (ACL):
Goran Glavaš and Ivan Vulić. 2021. Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3090–3104, Online. Association for Computational Linguistics.
Cite (Informal):
Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (Glavaš & Vulić, EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.270.pdf
Software:
 2021.eacl-main.270.Software.zip
Data
COPAGLUEMultiNLIPAWSPAWS-XUniversal DependenciesXCOPAXNLI