HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew

Tzuf Paz-Argaman, Itai Mondshine, Asaf Achi Mordechai, Reut Tsarfaty


Abstract
While large language models (LLMs) excel in various natural language tasks in English, their performance in low-resource languages like Hebrew, especially for generative tasks such as abstractive summarization, remains unclear. The high morphological richness in Hebrew adds further challenges due to the ambiguity in sentence comprehension and the complexities in meaning construction.In this paper, we address this evaluation and resource gap by introducing HeSum, a novel benchmark dataset specifically designed for Hebrew abstractive text summarization. HeSum consists of 10,000 article-summary pairs sourced from Hebrew news websites written by professionals. Linguistic analysis confirms HeSum’s high abstractness and unique morphological challenges. We show that HeSum presents distinct difficulties even for state-of-the-art LLMs, establishing it as a valuable testbed for advancing generative language technology in Hebrew, and MRLs generative challenges in general.
Anthology ID:
2024.findings-acl.381
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6378–6388
Language:
URL:
https://aclanthology.org/2024.findings-acl.381
DOI:
Bibkey:
Cite (ACL):
Tzuf Paz-Argaman, Itai Mondshine, Asaf Achi Mordechai, and Reut Tsarfaty. 2024. HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew. In Findings of the Association for Computational Linguistics ACL 2024, pages 6378–6388, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew (Paz-Argaman et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.381.pdf