Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT

Karim Lasri, Olga Seminck, Alessandro Lenci, Thierry Poibeau


Abstract
Both humans and neural language models are able to perform subject verb number agreement (SVA). In principle, semantics shouldn’t interfere with this task, which only requires syntactic knowledge. In this work we test whether meaning interferes with this type of agreement in English in syntactic structures of various complexities. To do so, we generate both semantically well-formed and nonsensical items. We compare the performance of BERT-base to that of humans, obtained with a psycholinguistic online crowdsourcing experiment. We find that BERT and humans are both sensitive to our semantic manipulation: They fail more often when presented with nonsensical items, especially when their syntactic structure features an attractor (a noun phrase between the subject and the verb that has not the same number as the subject). We also find that the effect of meaningfulness on SVA errors is stronger for BERT than for humans, showing higher lexical sensitivity of the former on this task.
Anthology ID:
2022.coling-1.4
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Editors:
Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
37–43
Language:
URL:
https://aclanthology.org/2022.coling-1.4
DOI:
Bibkey:
Cite (ACL):
Karim Lasri, Olga Seminck, Alessandro Lenci, and Thierry Poibeau. 2022. Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT. In Proceedings of the 29th International Conference on Computational Linguistics, pages 37–43, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT (Lasri et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.4.pdf