Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers

Aditya Yedetore, Najoung Kim


Abstract
Neural networks without hierarchical biases often struggle to learn linguistic rules that come naturally to humans. However, neural networks are trained primarily on form alone, while children acquiring language additionally receive data about meaning. Would neural networks generalize more like humans when trained on both form and meaning? We investigate this by examining if Transformers—neural networks without a hierarchical bias—better achieve hierarchical generalization when trained on both form and meaning compared to when trained on form alone. Our results show that Transformers trained on form and meaning do favor the hierarchical generalization more than those trained on form alone, suggesting that statistical learners without hierarchical biases can leverage semantic training signals to bootstrap hierarchical syntactic generalization.
Anthology ID:
2024.emnlp-main.235
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4059–4073
Language:
URL:
https://aclanthology.org/2024.emnlp-main.235
DOI:
10.18653/v1/2024.emnlp-main.235
Bibkey:
Cite (ACL):
Aditya Yedetore and Najoung Kim. 2024. Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4059–4073, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers (Yedetore & Kim, EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.235.pdf