PaLM: A Hybrid Parser and Language Model

Hao Peng, Roy Schwartz, Noah A. Smith


Abstract
We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.
Anthology ID:
D19-1376
Volume:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Kentaro Inui, Jing Jiang, Vincent Ng, Xiaojun Wan
Venues:
EMNLP | IJCNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3644–3651
Language:
URL:
https://aclanthology.org/D19-1376
DOI:
10.18653/v1/D19-1376
Bibkey:
Cite (ACL):
Hao Peng, Roy Schwartz, and Noah A. Smith. 2019. PaLM: A Hybrid Parser and Language Model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3644–3651, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
PaLM: A Hybrid Parser and Language Model (Peng et al., EMNLP-IJCNLP 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-1376.pdf
Attachment:
 D19-1376.Attachment.pdf
Code
 Noahs-ARK/PaLM
Data
Penn TreebankWikiText-2