The Pure Poet: How Good is the Subjective Credibility and Stylistic Quality of Literary Short Texts Written with an Artificial Intelligence Tool as Compared to Texts Written by Human Authors?

Vivian Emily Gunser, Steffen Gottschling, Birgit Brucker, Sandra Richter, Dîlan Canan Çakir, Peter Gerjets


Abstract
The application of artificial intelligence (AI) for text generation in creative domains raises questions regarding the credibility of AI-generated content. In two studies, we explored if readers can differentiate between AI-based and human-written texts (generated based on the first line of texts and poems of classic authors) and how the stylistic qualities of these texts are rated. Participants read 9 AI-based continuations and either 9 human-written continuations (Study 1, N=120) or 9 original continuations (Study 2, N=302). Participants’ task was to decide whether a continuation was written with an AI-tool or not, to indicate their confidence in each decision, and to assess the stylistic text quality. Results showed that participants generally had low accuracy for differentiating between text types but were overconfident in their decisions. Regarding the assessment of stylistic quality, AI-continuations were perceived as less well-written, inspiring, fascinating, interesting, and aesthetic than both human-written and original continuations.
Anthology ID:
2022.in2writing-1.8
Volume:
Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Ting-Hao 'Kenneth' Huang, Vipul Raheja, Dongyeop Kang, John Joon Young Chung, Daniel Gissin, Mina Lee, Katy Ilonka Gero
Venue:
In2Writing
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
60–61
Language:
URL:
https://aclanthology.org/2022.in2writing-1.8
DOI:
10.18653/v1/2022.in2writing-1.8
Bibkey:
Cite (ACL):
Vivian Emily Gunser, Steffen Gottschling, Birgit Brucker, Sandra Richter, Dîlan Canan Çakir, and Peter Gerjets. 2022. The Pure Poet: How Good is the Subjective Credibility and Stylistic Quality of Literary Short Texts Written with an Artificial Intelligence Tool as Compared to Texts Written by Human Authors?. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 60–61, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
The Pure Poet: How Good is the Subjective Credibility and Stylistic Quality of Literary Short Texts Written with an Artificial Intelligence Tool as Compared to Texts Written by Human Authors? (Gunser et al., In2Writing 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.in2writing-1.8.pdf
Video:
 https://aclanthology.org/2022.in2writing-1.8.mp4