Subjective and Hierarchical Annotation of Text Difficulty in Japanese

Daisuke Maekawa, Kazumasa Omura, Sora Tarumoto, Shotaro Ishihara, Tomoyuki Kajiwara


Abstract
This study presents a difficulty-annotated corpus of Japanese newspaper articles. The annotation in our corpus has two key features. First, it is hierarchical, with difficulty levels labeled at the word, sentence, and document levels, respectively. Second, it is subjective, with 99 readers from diverse backgrounds labeling the difficulty levels based on their own perceptions. Detailed analysis of our corpus revealed correlations in difficulty levels across linguistic units, such as between words and sentences, and sentences and documents. We also found that text difficulty varies depending on reader attributes such as age and topics of interest. Furthermore, we trained Transformer encoders on this corpus and developed a personalized difficulty estimation model that takes into account both linguistic and attribute information. Experimental results demonstrate that incorporating reader attributes improves the performance of difficulty estimation and that multi-task learning across language units is beneficial. Our difficulty annotations are publicly available.
Anthology ID:
2026.emnlp-main.926
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
21316–21327
Language:
URL:
https://aclanthology.org/2026.emnlp-main.926/
DOI:
Bibkey:
Cite (ACL):
Daisuke Maekawa, Kazumasa Omura, Sora Tarumoto, Shotaro Ishihara, and Tomoyuki Kajiwara. 2026. Subjective and Hierarchical Annotation of Text Difficulty in Japanese. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 21316–21327, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Subjective and Hierarchical Annotation of Text Difficulty in Japanese (Maekawa et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.926.pdf