BLM-AgrF: A New French Benchmark to Investigate Generalization of Agreement in Neural Networks

Aixiu An, Chunyang Jiang, Maria A. Rodriguez, Vivi Nastase, Paola Merlo


Abstract
Successful machine learning systems currently rely on massive amounts of data, which are very effective in hiding some of the shallowness of the learned models. To help train models with more complex and compositional skills, we need challenging data, on which a system is successful only if it detects structure and regularities, that will allow it to generalize. In this paper, we describe a French dataset (BLM-AgrF) for learning the underlying rules of subject-verb agreement in sentences, developed in the BLM framework, a new task inspired by visual IQ tests known as Raven’s Progressive Matrices. In this task, an instance consists of sequences of sentences with specific attributes. To predict the correct answer as the next element of the sequence, a model must correctly detect the generative model used to produce the dataset. We provide details and share a dataset built following this methodology. Two exploratory baselines based on commonly used architectures show that despite the simplicity of the phenomenon, it is a complex problem for deep learning systems.
Anthology ID:
2023.eacl-main.99
Volume:
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1363–1374
Language:
URL:
https://aclanthology.org/2023.eacl-main.99
DOI:
10.18653/v1/2023.eacl-main.99
Bibkey:
Cite (ACL):
Aixiu An, Chunyang Jiang, Maria A. Rodriguez, Vivi Nastase, and Paola Merlo. 2023. BLM-AgrF: A New French Benchmark to Investigate Generalization of Agreement in Neural Networks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1363–1374, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
BLM-AgrF: A New French Benchmark to Investigate Generalization of Agreement in Neural Networks (An et al., EACL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.eacl-main.99.pdf
Video:
 https://aclanthology.org/2023.eacl-main.99.mp4