How Much Syntactic Supervision is “Good Enough”?

Hiroshi Noji, Yohei Oseki


Abstract
In this paper, we explore how much syntactic supervision is “good enough” to make language models (LMs) more human-like. Specifically, we propose the new method called syntactic ablation, where syntactic LMs, namely Recurrent Neural Network Grammars (RNNGs), are gradually ablated from full syntactic supervision to zero syntactic supervision (≈ unidirectional LSTM) by preserving NP, VP, PP, SBAR nonterminal symbols and the combinations thereof. The 17 ablated grammars are then evaluated via targeted syntactic evaluation on the SyntaxGym benchmark. The results of our syntactic ablation demonstrated that (i) the RNNG with zero syntactic supervision underperformed the RNNGs with some syntactic supervision, (ii) the RNNG with full syntactic supervision underperformed the RNNGs with less syntactic supervision, and (iii) the RNNG with mild syntactic supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. Those results may suggest that the “good enough” approach to language processing seems to make LMs more human-like.
Anthology ID:
2023.findings-eacl.173
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2300–2305
Language:
URL:
https://aclanthology.org/2023.findings-eacl.173
DOI:
10.18653/v1/2023.findings-eacl.173
Bibkey:
Cite (ACL):
Hiroshi Noji and Yohei Oseki. 2023. How Much Syntactic Supervision is “Good Enough”?. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2300–2305, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
How Much Syntactic Supervision is “Good Enough”? (Noji & Oseki, Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.173.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.173.mp4