Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods

Tatiana Passali, Grigorios Tsoumakas


Abstract
Topic-controllable summarization is an emerging research area with a wide range of potential applications. However, existing approaches suffer from significant limitations. For example, the majority of existing methods built upon recurrent architectures, which can significantly limit their performance compared to more recent Transformer-based architectures, while they also require modifications to the model’s architecture for controlling the topic. At the same time, there is currently no established evaluation metric designed specifically for topic-controllable summarization. This work proposes a new topic-oriented evaluation measure to automatically evaluate the generated summaries based on the topic affinity between the generated summary and the desired topic. The reliability of the proposed measure is demonstrated through appropriately designed human evaluation. In addition, we adapt topic embeddings to work with powerful Transformer architectures and propose a novel and efficient approach for guiding the summary generation through control tokens. Experimental results reveal that control tokens can achieve better performance compared to more complicated embedding-based approaches while also being significantly faster.
Anthology ID:
2024.lrec-main.1415
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
16282–16292
Language:
URL:
https://aclanthology.org/2024.lrec-main.1415
DOI:
Bibkey:
Cite (ACL):
Tatiana Passali and Grigorios Tsoumakas. 2024. Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16282–16292, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods (Passali & Tsoumakas, LREC-COLING 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.lrec-main.1415.pdf