Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing

Ran Ji, Jianmin Ji


Abstract
Semantic parsing considers the task of mapping a natural language sentence into a target formal representation, where various sophisticated sequence-to-sequence (seq2seq) models have been applied with promising results. Generally, these target representations follow a syntax formalism that limits permitted forms. However, it is neither easy nor flexible to explicitly integrate this syntax formalism into a neural seq2seq model. In this paper, we present a structure-aware self-attention language model to capture structural information of target representations and propose a knowledge distillation based approach to incorporating the target language model into a seq2seq model, where grammar rules or sketches are not required in the training process. An ablation study shows that the proposed language model can notably improve the performance of the baseline model. The experiments show that our method achieves new state-of-the-art performance among neural approaches on four semantic parsing (ATIS, GEO) and Python code generation (Django, CoNaLa) tasks.
Anthology ID:
2022.coling-1.280
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3164–3174
Language:
URL:
https://aclanthology.org/2022.coling-1.280
DOI:
Bibkey:
Cite (ACL):
Ran Ji and Jianmin Ji. 2022. Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3164–3174, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing (Ji & Ji, COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.280.pdf
Data
CoNaLaDjango