Actor Identification in Discourse: A Challenge for LLMs?

Ana Barić, Sebastian Padó, Sean Papay


Abstract
The identification of political actors who put forward claims in public debate is a crucial step in the construction of discourse networks, which are helpful to analyze societal debates. Actor identification is, however, rather challenging: Often, the locally mentioned speaker of a claim is only a pronoun (“He proposed that [claim]”), so recovering the canonical actor name requires discourse understanding. We compare a traditional pipeline of dedicated NLP components (similar to those applied to the related task of coreference) with a LLM, which appears a good match for this generation task. Evaluating on a corpus of German actors in newspaper reports, we find surprisingly that the LLM performs worse. Further analysis reveals that the LLM is very good at identifying the right reference, but struggles to generate the correct canonical form. This points to an underlying issue in LLMs with controlling generated output. Indeed, a hybrid model combining the LLM with a classifier to normalize its output substantially outperforms both initial models.
Anthology ID:
2024.codi-1.6
Volume:
Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024)
Month:
March
Year:
2024
Address:
St. Julians, Malta
Editors:
Michael Strube, Chloe Braud, Christian Hardmeier, Junyi Jessy Li, Sharid Loaiciga, Amir Zeldes, Chuyuan Li
Venues:
CODI | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
64–70
Language:
URL:
https://aclanthology.org/2024.codi-1.6
DOI:
Bibkey:
Cite (ACL):
Ana Barić, Sebastian Padó, and Sean Papay. 2024. Actor Identification in Discourse: A Challenge for LLMs?. In Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024), pages 64–70, St. Julians, Malta. Association for Computational Linguistics.
Cite (Informal):
Actor Identification in Discourse: A Challenge for LLMs? (Barić et al., CODI-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.codi-1.6.pdf
Supplementary material:
 2024.codi-1.6.SupplementaryMaterial.gz
Video:
 https://aclanthology.org/2024.codi-1.6.mp4