Language Agnostic Automatic Summarization Evaluation

Christopher Tauchmann, Margot Mieskes


Abstract
So far work on automatic summarization has dealt primarily with English data. Accordingly, evaluation methods were primarily developed with this language in mind. In our work, we present experiments of adapting available evaluation methods such as ROUGE and PYRAMID to non-English data. We base our experiments on various English and non-English homogeneous benchmark data sets as well as a non-English heterogeneous data set. Our results indicate that ROUGE can indeed be adapted to non-English data – both homogeneous and heterogeneous. Using a recent implementation of performing an automatic PYRAMID evaluation, we also show its adaptability to non-English data.
Anthology ID:
2020.lrec-1.822
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
6656–6662
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.822
DOI:
Bibkey:
Cite (ACL):
Christopher Tauchmann and Margot Mieskes. 2020. Language Agnostic Automatic Summarization Evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6656–6662, Marseille, France. European Language Resources Association.
Cite (Informal):
Language Agnostic Automatic Summarization Evaluation (Tauchmann & Mieskes, LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.822.pdf