How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks

Bingzhi Li, Guillaume Wisniewski, Benoit Crabbé


Abstract
This work addresses the question of the localization of syntactic information encoded in the transformers representations. We tackle this question from two perspectives, considering the object-past participle agreement in French, by identifying, first, in which part of the sentence and, second, in which part of the representation the syntactic information is encoded. The results of our experiments, using probing, causal analysis and feature selection method, show that syntactic information is encoded locally in a way consistent with the French grammar.
Anthology ID:
2022.acl-short.54
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
501–507
Language:
URL:
https://aclanthology.org/2022.acl-short.54
DOI:
10.18653/v1/2022.acl-short.54
Bibkey:
Cite (ACL):
Bingzhi Li, Guillaume Wisniewski, and Benoit Crabbé. 2022. How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 501–507, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks (Li et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-short.54.pdf
Video:
 https://aclanthology.org/2022.acl-short.54.mp4