Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks

Yichen Jiang, Mohit Bansal


Abstract
Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. However, existing neural models have been shown to lack this basic ability in learning symbolic structures. Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks as additional training supervision. These automatically-generated sequences are more representative of the underlying compositional symbolic structures of the input data. During inference, the model jointly predicts the next action and the next tokens in the auxiliary sequences at each step. Experiments on the SCAN dataset show that our method encourages the Transformer to understand compositional structures of the command, improving its accuracy on multiple challenging splits from ≤ 10% to 100%. With only 418 (5%) training instances, our approach still achieves 97.8% accuracy on the MCD1 split. Therefore, we argue that compositionality can be induced in Transformers given minimal but proper guidance. We also show that a better result is achieved using less contextualized vectors as the attention’s query, providing insights into architecture choices in achieving systematic compositionality. Finally, we show positive generalization results on the grounded-SCAN task (Ruis et al., 2020).
Anthology ID:
2021.emnlp-main.505
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6253–6265
Language:
URL:
https://aclanthology.org/2021.emnlp-main.505
DOI:
10.18653/v1/2021.emnlp-main.505
Bibkey:
Cite (ACL):
Yichen Jiang and Mohit Bansal. 2021. Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6253–6265, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks (Jiang & Bansal, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.505.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.505.mp4
Code
 jiangyctarheel/compositional-auxseq
Data
CFQSCAN