AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content

Shuyang Cao, Lu Wang


Abstract
Long document summarization systems are critical for domains with lengthy and jargon-laden text, yet they present significant challenges to researchers and developers with limited computing resources. Existing solutions mainly focus on efficient attentions or divide-and-conquer strategies. The former reduces theoretical time complexity, but is still memory-heavy. The latter methods sacrifice global context, leading to uninformative and incoherent summaries. This work aims to leverage the memory-efficient nature of divide-and-conquer methods while preserving global context. Concretely, our framework AWESOME uses two novel mechanisms: (1) External memory mechanisms track previously encoded document segments and their corresponding summaries, to enhance global document understanding and summary coherence. (2) Global salient content is further identified beforehand to augment each document segment to support its summarization. Extensive experiments on diverse genres of text, including government reports, meeting transcripts, screenplays, scientific papers, and novels, show that AWESOME produces summaries with improved informativeness, faithfulness, and coherence than competitive baselines on longer documents, while having a smaller GPU memory footprint.
Anthology ID:
2024.naacl-long.330
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5925–5941
Language:
URL:
https://aclanthology.org/2024.naacl-long.330
DOI:
Bibkey:
Cite (ACL):
Shuyang Cao and Lu Wang. 2024. AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5925–5941, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content (Cao & Wang, NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.330.pdf
Copyright:
 2024.naacl-long.330.copyright.pdf