Structural generalization is hard for sequence-to-sequence models

Yuekun Yao, Alexander Koller


Abstract
Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks,including ones that require predicting linguistic structure. However, recent work on compositional generalization has shown that seq2seq models achieve very low accuracy in generalizing to linguistic structures that were not seen in training. We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in. We further report on some experiments that give initial answers on the reasons for these limitations.
Anthology ID:
2022.emnlp-main.337
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5048–5062
Language:
URL:
https://aclanthology.org/2022.emnlp-main.337
DOI:
10.18653/v1/2022.emnlp-main.337
Bibkey:
Cite (ACL):
Yuekun Yao and Alexander Koller. 2022. Structural generalization is hard for sequence-to-sequence models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048–5062, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Structural generalization is hard for sequence-to-sequence models (Yao & Koller, EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.337.pdf