Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings

Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, Fang Chen


Abstract
We present a new summary evaluation approach that does not require human model summaries. Our approach exploits the compositional capabilities of corpus-based and lexical resource-based word embeddings to develop the features reflecting coverage, diversity, informativeness, and coherence of summaries. The features are then used to train a learning model for predicting the summary content quality in the absence of gold models. We evaluate the proposed metric in replicating the human assigned scores for summarization systems and summaries on data from query-focused and update summarization tasks in TAC 2008 and 2009. The results show that our feature combination provides reliable estimates of summary content quality when model summaries are not available.
Anthology ID:
C18-1077
Volume:
Proceedings of the 27th International Conference on Computational Linguistics
Month:
August
Year:
2018
Address:
Santa Fe, New Mexico, USA
Editors:
Emily M. Bender, Leon Derczynski, Pierre Isabelle
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
905–914
Language:
URL:
https://aclanthology.org/C18-1077
DOI:
Bibkey:
Cite (ACL):
Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, and Fang Chen. 2018. Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings. In Proceedings of the 27th International Conference on Computational Linguistics, pages 905–914, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Cite (Informal):
Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (ShafieiBavani et al., COLING 2018)
Copy Citation:
PDF:
https://aclanthology.org/C18-1077.pdf