When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes

Mycal Tucker, Tiwalayo Eisape, Peng Qian, Roger Levy, Julie Shah


Abstract
Recent causal probing literature reveals when language models and syntactic probes use similar representations. Such techniques may yield “false negative” causality results: models may use representations of syntax, but probes may have learned to use redundant encodings of the same syntactic information. We demonstrate that models do encode syntactic information redundantly and introduce a new probe design that guides probes to consider all syntactic information present in embeddings. Using these probes, we find evidence for the use of syntax in models where prior methods did not, allowing us to boost model performance by injecting syntactic information into representations.
Anthology ID:
2022.naacl-main.394
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5393–5408
Language:
URL:
https://aclanthology.org/2022.naacl-main.394
DOI:
10.18653/v1/2022.naacl-main.394
Bibkey:
Cite (ACL):
Mycal Tucker, Tiwalayo Eisape, Peng Qian, Roger Levy, and Julie Shah. 2022. When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5393–5408, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes (Tucker et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.394.pdf
Data
MultiNLIPenn Treebank