Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers

Zhuang Li, Lizhen Qu, Gholamreza Haffari


Abstract
This paper investigates continual learning for semantic parsing. In this setting, a neural semantic parser learns tasks sequentially without accessing full training data from previous tasks. Direct application of the SOTA continual learning algorithms to this problem fails to achieve comparable performance with re-training models with all seen tasks because they have not considered the special properties of structured outputs yielded by semantic parsers. Therefore, we propose TotalRecall, a continual learning method designed for neural semantic parsers from two aspects: i) a sampling method for memory replay that diversifies logical form templates and balances distributions of parse actions in a memory; ii) a two-stage training method that significantly improves generalization capability of the parsers across tasks. We conduct extensive experiments to study the research problems involved in continual semantic parsing and demonstrate that a neural semantic parser trained with TotalRecall achieves superior performance than the one trained directly with the SOTA continual learning algorithms and achieve a 3-6 times speedup compared to re-training from scratch.
Anthology ID:
2021.emnlp-main.310
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3816–3831
Language:
URL:
https://aclanthology.org/2021.emnlp-main.310
DOI:
10.18653/v1/2021.emnlp-main.310
Bibkey:
Cite (ACL):
Zhuang Li, Lizhen Qu, and Gholamreza Haffari. 2021. Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3816–3831, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (Li et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.310.pdf
Code
 zhuang-li/cl_nsp