Generalizable Sarcasm Detection is Just Around the Corner, of Course!

Hyewon Jang, Diego Frassinelli


Abstract
We tested the robustness of sarcasm detection models by examining their behavior when fine-tuned on four sarcasm datasets containing varying characteristics of sarcasm: label source (authors vs. third-party), domain (social media/online vs. offline conversations/dialogues), style (aggressive vs. humorous mocking). We tested their prediction performance on the same dataset (intra-dataset) and across different datasets (cross-dataset). For intra-dataset predictions, models consistently performed better when fine-tuned with third-party labels rather than with author labels. For cross-dataset predictions, most models failed to generalize well to the other datasets, implying that one type of dataset cannot represent all sorts of sarcasm with different styles and domains. Compared to the existing datasets, models fine-tuned on the new dataset we release in this work showed the highest generalizability to other datasets. With a manual inspection of the datasets and post-hoc analysis, we attributed the difficulty in generalization to the fact that sarcasm actually comes in different domains and styles. We argue that future sarcasm research should take the broad scope of sarcasm into account.
Anthology ID:
2024.naacl-long.238
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4238–4249
Language:
URL:
https://aclanthology.org/2024.naacl-long.238
DOI:
Bibkey:
Cite (ACL):
Hyewon Jang and Diego Frassinelli. 2024. Generalizable Sarcasm Detection is Just Around the Corner, of Course!. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4238–4249, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Generalizable Sarcasm Detection is Just Around the Corner, of Course! (Jang & Frassinelli, NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.238.pdf
Copyright:
 2024.naacl-long.238.copyright.pdf