Efficient Few-Shot Fine-Tuning for Opinion Summarization

Arthur Brazinskas, Ramesh Nallapati, Mohit Bansal, Markus Dreyer


Abstract
Abstractive summarization models are typically pre-trained on large amounts of generic texts, then fine-tuned on tens or hundreds of thousands of annotated samples. However, in opinion summarization, large annotated datasets of reviews paired with reference summaries are not available and would be expensive to create. This calls for fine-tuning methods robust to overfitting on small datasets. In addition, generically pre-trained models are often not accustomed to the specifics of customer reviews and, after fine-tuning, yield summaries with disfluencies and semantic mistakes. To address these problems, we utilize an efficient few-shot method based on adapters which, as we show, can easily store in-domain knowledge. Instead of fine-tuning the entire model, we add adapters and pre-train them in a task-specific way on a large corpus of unannotated customer reviews, using held-out reviews as pseudo summaries. Then, fine-tune the adapters on the small available human-annotated dataset. We show that this self-supervised adapter pre-training improves summary quality over standard fine-tuning by 2.0 and 1.3 ROUGE-L points on the Amazon and Yelp datasets, respectively. Finally, for summary personalization, we condition on aspect keyword queries, automatically created from generic datasets. In the same vein, we pre-train the adapters in a query-based manner on customer reviews and then fine-tune them on annotated datasets. This results in better-organized summary content reflected in improved coherence and fewer redundancies.
Anthology ID:
2022.findings-naacl.113
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1509–1523
Language:
URL:
https://aclanthology.org/2022.findings-naacl.113
DOI:
10.18653/v1/2022.findings-naacl.113
Bibkey:
Cite (ACL):
Arthur Brazinskas, Ramesh Nallapati, Mohit Bansal, and Markus Dreyer. 2022. Efficient Few-Shot Fine-Tuning for Opinion Summarization. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1509–1523, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Efficient Few-Shot Fine-Tuning for Opinion Summarization (Brazinskas et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-naacl.113.pdf
Video:
 https://aclanthology.org/2022.findings-naacl.113.mp4
Code
 amazon-research/adasum