SCRIPT: Self-Critic PreTraining of Transformers

Erik Nijkamp, Bo Pang, Ying Nian Wu, Caiming Xiong


Abstract
We introduce Self-CRItic Pretraining Transformers (SCRIPT) for representation learning of text. The popular masked language modeling (MLM) pretraining methods like BERT replace some tokens with [MASK] and an encoder is trained to recover them, while ELECTRA trains a discriminator to detect replaced tokens proposed by a generator. In contrast, we train a language model as in MLM and further derive a discriminator or critic on top of the encoder without using any additional parameters. That is, the model itself is a critic. SCRIPT combines MLM training and discriminative training for learning rich representations and compute- and sample-efficiency. We demonstrate improved sample-efficiency in pretraining and enhanced representations evidenced by improved downstream task performance on GLUE and SQuAD over strong baselines. Also, the self-critic scores can be directly used as pseudo-log-likelihood for efficient scoring.
Anthology ID:
2021.naacl-main.409
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5196–5202
Language:
URL:
https://aclanthology.org/2021.naacl-main.409
DOI:
10.18653/v1/2021.naacl-main.409
Bibkey:
Cite (ACL):
Erik Nijkamp, Bo Pang, Ying Nian Wu, and Caiming Xiong. 2021. SCRIPT: Self-Critic PreTraining of Transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5196–5202, Online. Association for Computational Linguistics.
Cite (Informal):
SCRIPT: Self-Critic PreTraining of Transformers (Nijkamp et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.409.pdf
Video:
 https://aclanthology.org/2021.naacl-main.409.mp4
Data
GLUEOpenWebTextSQuADWebText