Neural Network Acceptability Judgments

Alex Warstadt, Amanpreet Singh, Samuel R. Bowman


Abstract
This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.
Anthology ID:
Q19-1040
Volume:
Transactions of the Association for Computational Linguistics, Volume 7
Month:
Year:
2019
Address:
Cambridge, MA
Editors:
Lillian Lee, Mark Johnson, Brian Roark, Ani Nenkova
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
625–641
Language:
URL:
https://aclanthology.org/Q19-1040
DOI:
10.1162/tacl_a_00290
Bibkey:
Cite (ACL):
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7:625–641.
Cite (Informal):
Neural Network Acceptability Judgments (Warstadt et al., TACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/Q19-1040.pdf
Code
 additional community code
Data
CoLAGLUE