Creation of an Evaluation Corpus and Baseline Evaluation Scores for Welsh Text Summarisation

Mahmoud El-Haj, Ignatius Ezeani, Jonathan Morris, Dawn Knight


Abstract
As part of the effort to increase the availability of Welsh digital technology, this paper introduces the first human vs metrics Welsh summarisation evaluation results and dataset, which we provide freely for research purposes to help advance the work on Welsh summarisation. The system summaries were created using an extractive graph-based Welsh summariser. The system summaries were evaluated by both human and a range of ROUGE metric variants (e.g. ROUGE 1, 2, L and SU4). The summaries and evaluation results will serve as benchmarks for the development of summarisers and evaluation metrics in other minority language contexts.
Anthology ID:
2022.cltw-1.3
Volume:
Proceedings of the 4th Celtic Language Technology Workshop within LREC2022
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Theodorus Fransen, William Lamb, Delyth Prys
Venue:
CLTW
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
14–21
Language:
URL:
https://aclanthology.org/2022.cltw-1.3
DOI:
Bibkey:
Cite (ACL):
Mahmoud El-Haj, Ignatius Ezeani, Jonathan Morris, and Dawn Knight. 2022. Creation of an Evaluation Corpus and Baseline Evaluation Scores for Welsh Text Summarisation. In Proceedings of the 4th Celtic Language Technology Workshop within LREC2022, pages 14–21, Marseille, France. European Language Resources Association.
Cite (Informal):
Creation of an Evaluation Corpus and Baseline Evaluation Scores for Welsh Text Summarisation (El-Haj et al., CLTW 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.cltw-1.3.pdf