LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

Ruizhi Zhang, Jinwei Chen, Lu Xiangju, He Yan, Mo Yu, Junmin Zhu, Wei Zhang


Abstract
Although context windows have expanded significantly in recent years, hallucinations in long-context summarization remain a challenge. Long novels are better suited than news or papers for researching these hallucinations, due to their intrinsic information and detailed descriptions of events and dialogues. However, current research lacks a multi-scale benchmark for hallucination detection in long-context novel summarization and does not fully explore how hallucinations change as the context grows longer. In this study, we propose LongNovel, a multi-scale long-context bilingual (Chinese and English) novel benchmark for hallucination detection. This benchmark is constructed from 29 Chinese novels (ranging from 16k to 100k tokens) and chapter-level data from the BookSum dataset. We design 8 hallucination types and employ a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation to ensure both data authenticity and a balanced distribution of hallucination categories. Furthermore, we manually revise the content in the test set to guarantee data reliability. Extensive experimental results demonstrate that LongNovel is a challenging benchmark. We release LongNovel for future research. https://github.com/BDML-lab/LongNove
Anthology ID:
2026.emnlp-main.1423
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
32478–32538
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1423/
DOI:
Bibkey:
Cite (ACL):
Ruizhi Zhang, Jinwei Chen, Lu Xiangju, He Yan, Mo Yu, Junmin Zhu, and Wei Zhang. 2026. LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 32478–32538, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization (Zhang et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1423.pdf