Yinan Hu
2022
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization
Xiaochen Liu
|
Yang Gao
|
Yu Bai
|
Jiawei Li
|
Yinan Hu
|
Heyan Huang
|
Boxing Chen
Proceedings of the 29th International Conference on Computational Linguistics
Few-shot abstractive summarization has become a challenging task in natural language generation. To support it, we developed a novel soft prompts architecture coupled with a prompt pre-training plus prompt fine-tuning paradigm, which is effective and tunes only extremely light parameters. To meet the structure of the generation models, the soft prompts comprise continuous input embeddings across an encoder and a decoder. Importantly, a new inner-prompt placed in the text is introduced to capture document-level information. The aim is to devote attention to understanding the document that better prompts the model to generate document-related content. In the training process, the prompt pre-training with self-supervised pseudo-data firstly teaches the model basic summarizing capability. Then, with few-shot examples, only the designed lightweight soft prompts are fine-tuned. Experimental results on the CNN/DailyMail and XSum datasets show that our method, with only 0.1% of the parameters, outperforms full-model tuning where all model parameters are tuned. It also surpasses Prompt Tuning by a large margin and delivers competitive results against Prefix-Tuning with 3% of the parameters.
Search
Co-authors
- Boxing Chen 1
- He-Yan Huang 1
- Jiawei Li 1
- Xiaochen Liu 1
- Yang Gao (扬 高) 1
- show all...
- Yu Bai 1