Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, yet they struggle with causal reasoning tasks. We introduce PACKaGE (Paragraph-level Annotated Causal Knowledge and Graph–tExt), a causal benchmark dataset that pairs paragraph-level texts with explicit causal graphs. Unlike prior sentence-level resources, our benchmark dataset (i) operates at paragraph level to capture various and interdependent causal pathways; (ii) offers multimodal annotations including full graphs and contextualized relations; and (iii) maps extended passages to comprehensive graphs. We detail a four-stage construction pipeline: collecting causality-labeled and fact-centric texts, inducing graphs from span-tagged pairs, condensing nodes, and recomposing path-aligned narratives. Overall, the paired evaluation highlights a persistent discrepancy between linguistic competence and causal understanding in LLMs, pointing to the need for further development.

Finding causal relations in texts has been a challenge since it requires methods ranging from defining event ontologies to developing proper algorithmic approaches. In this paper, we developed a framework which classifies whether a given sentence contains a causal event. As our approach, we exploited an external corpus that has causal labels to overcome the small size of the original corpus (Causal News Corpus) provided by task organizers. Further, we employed a data augmentation technique utilizing Part-Of-Speech (POS) based on our observation that some parts of speech are more (or less) relevant to causality. Our approach especially improved the recall of detecting causal events in sentences.