Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator

Xiaoyu Shen, Yang Zhao, Hui Su, Dietrich Klakow


Abstract
Pointer Generators have been the de facto standard for modern summarization systems. However, this architecture faces two major drawbacks: Firstly, the pointer is limited to copying the exact words while ignoring possible inflections or abstractions, which restricts its power of capturing richer latent alignment. Secondly, the copy mechanism results in a strong bias towards extractive generations, where most sentences are produced by simply copying from the source text. In this paper, we address these problems by allowing the model to “edit” pointed tokens instead of always hard copying them. The editing is performed by transforming the pointed word vector into a target space with a learned relation embedding. On three large-scale summarization dataset, we show the model is able to (1) capture more latent alignment relations than exact word matches, (2) improve word alignment accuracy, allowing for better model interpretation and controlling, (3) generate higher-quality summaries validated by both qualitative and quantitative evaluations and (4) bring more abstraction to the generated summaries.
Anthology ID:
D19-1390
Volume:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Kentaro Inui, Jing Jiang, Vincent Ng, Xiaojun Wan
Venues:
EMNLP | IJCNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3762–3773
Language:
URL:
https://aclanthology.org/D19-1390
DOI:
10.18653/v1/D19-1390
Bibkey:
Cite (ACL):
Xiaoyu Shen, Yang Zhao, Hui Su, and Dietrich Klakow. 2019. Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3762–3773, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator (Shen et al., EMNLP-IJCNLP 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-1390.pdf
Attachment:
 D19-1390.Attachment.pdf