Systematic Evaluation of a Framework for Unsupervised Emotion Recognition for Narrative Text

Samira Zad, Mark Finlayson


Abstract
Identifying emotions as expressed in text (a.k.a. text emotion recognition) has received a lot of attention over the past decade. Narratives often involve a great deal of emotional expression, and so emotion recognition on narrative text is of great interest to computational approaches to narrative understanding. Prior work by Kim et al. 2010 was the work with the highest reported emotion detection performance, on a corpus of fairy tales texts. Close inspection of that work, however, revealed significant reproducibility problems, and we were unable to reimplement Kim’s approach as described. As a consequence, we implemented a framework inspired by Kim’s approach, where we carefully evaluated the major design choices. We identify the highest-performing combination, which outperforms Kim’s reported performance by 7.6 F1 points on average. Close inspection of the annotated data revealed numerous missing and incorrect emotion terms in the relevant lexicon, WordNetAffect (WNA; Strapparava and Valitutti, 2004), which allowed us to augment it in a useful way. More generally, this showed that numerous clearly emotive words and phrases are missing from WNA, which suggests that effort invested in augmenting or refining emotion ontologies could be useful for improving the performance of emotion recognition systems. We release our code and data to definitely enable future reproducibility of this work.
Anthology ID:
2020.nuse-1.4
Volume:
Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events
Month:
July
Year:
2020
Address:
Online
Editors:
Claire Bonial, Tommaso Caselli, Snigdha Chaturvedi, Elizabeth Clark, Ruihong Huang, Mohit Iyyer, Alejandro Jaimes, Heng Ji, Lara J. Martin, Ben Miller, Teruko Mitamura, Nanyun Peng, Joel Tetreault
Venues:
NUSE | WNU
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
26–37
Language:
URL:
https://aclanthology.org/2020.nuse-1.4
DOI:
10.18653/v1/2020.nuse-1.4
Bibkey:
Cite (ACL):
Samira Zad and Mark Finlayson. 2020. Systematic Evaluation of a Framework for Unsupervised Emotion Recognition for Narrative Text. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 26–37, Online. Association for Computational Linguistics.
Cite (Informal):
Systematic Evaluation of a Framework for Unsupervised Emotion Recognition for Narrative Text (Zad & Finlayson, NUSE-WNU 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.nuse-1.4.pdf
Video:
 http://slideslive.com/38929743