The Empirical Variability of Narrative Perceptions of Social Media Texts

Joel Mire, Maria Antoniak, Elliott Ash, Andrew Piper, Maarten Sap


Abstract
Most NLP work on narrative detection has focused on prescriptive definitions of stories crafted by researchers, leaving open the questions: how do crowd workers perceive texts to be a story, and why? We investigate this by building StoryPerceptions, a dataset of 2,496 perceptions of storytelling in 502 social media texts from 255 crowd workers, including categorical labels along with free-text storytelling rationales, authorial intent, and more. We construct a fine-grained bottom-up taxonomy of crowd workers’ varied and nuanced perceptions of storytelling by open-coding their free-text rationales. Through comparative analyses at the label and code level, we illuminate patterns of disagreement among crowd workers and across other annotation contexts, including prescriptive labeling from researchers and LLM-based predictions. Notably, plot complexity, references to generalized or abstract actions, and holistic aesthetic judgments (such as a sense of cohesion) are especially important in disagreements. Our empirical findings broaden understanding of the types, relative importance, and contentiousness of features relevant to narrative detection, highlighting opportunities for future work on reader-contextualized models of narrative reception.
Anthology ID:
2024.emnlp-main.1113
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
19940–19968
Language:
URL:
https://aclanthology.org/2024.emnlp-main.1113
DOI:
Bibkey:
Cite (ACL):
Joel Mire, Maria Antoniak, Elliott Ash, Andrew Piper, and Maarten Sap. 2024. The Empirical Variability of Narrative Perceptions of Social Media Texts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19940–19968, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
The Empirical Variability of Narrative Perceptions of Social Media Texts (Mire et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.1113.pdf