Structural Guidance for Transformer Language Models

Peng Qian, Tahira Naseem, Roger Levy, Ramón Fernandez Astudillo


Abstract
Transformer-based language models pre-trained on large amounts of text data have proven remarkably successful in learning generic transferable linguistic representations. Here we study whether structural guidance leads to more human-like systematic linguistic generalization in Transformer language models without resorting to pre-training on very large amounts of data. We explore two general ideas. The “Generative Parsing” idea jointly models the incremental parse and word sequence as part of the same sequence modeling task. The “Structural Scaffold” idea guides the language model’s representation via additional structure loss that separately predicts the incremental constituency parse. We train the proposed models along with a vanilla Transformer language model baseline on a 14 million-token and a 46 million-token subset of the BLLIP dataset, and evaluate models’ syntactic generalization performances on SG Test Suites and sized BLiMP. Experiment results across two benchmarks suggest converging evidence that generative structural supervisions can induce more robust and humanlike linguistic generalization in Transformer language models without the need for data intensive pre-training.
Anthology ID:
2021.acl-long.289
Original:
2021.acl-long.289v1
Version 2:
2021.acl-long.289v2
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3735–3745
Language:
URL:
https://aclanthology.org/2021.acl-long.289
DOI:
10.18653/v1/2021.acl-long.289
Bibkey:
Cite (ACL):
Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernandez Astudillo. 2021. Structural Guidance for Transformer Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3735–3745, Online. Association for Computational Linguistics.
Cite (Informal):
Structural Guidance for Transformer Language Models (Qian et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.289.pdf
Video:
 https://aclanthology.org/2021.acl-long.289.mp4
Code
 IBM/transformers-struct-guidance
Data
BLiMP