Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue

Alexios Gidiotis, Grigorios Tsoumakas


Abstract
We explore the notion of uncertainty in the context of modern abstractive summarization models, using the tools of Bayesian Deep Learning. Our approach approximates Bayesian inference by first extending state-of-the-art summarization models with Monte Carlo dropout and then using them to perform multiple stochastic forward passes. Based on Bayesian inference we are able to effectively quantify uncertainty at prediction time. Having a reliable uncertainty measure, we can improve the experience of the end user by filtering out generated summaries of high uncertainty. Furthermore, uncertainty estimation could be used as a criterion for selecting samples for annotation, and can be paired nicely with active learning and human-in-the-loop approaches. Finally, Bayesian inference enables us to find a Bayesian summary which performs better than a deterministic one and is more robust to uncertainty. In practice, we show that our Variational Bayesian equivalents of BART and PEGASUS can outperform their deterministic counterparts on multiple benchmark datasets.
Anthology ID:
2022.findings-acl.325
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venues:
ACL | Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4119–4131
Language:
URL:
https://aclanthology.org/2022.findings-acl.325
DOI:
10.18653/v1/2022.findings-acl.325
Bibkey:
Cite (ACL):
Alexios Gidiotis and Grigorios Tsoumakas. 2022. Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4119–4131, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue (Gidiotis & Tsoumakas, Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.325.pdf
Data
AESLC