Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation

Zijian Ding, Alison Smith-Renner, Wenjuan Zhang, Joel Tetreault, Alejandro Jaimes


Abstract
To explore how humans can best leverage LLMs for writing and how interacting with these models affects feelings of ownership and trust in the writing process, we compared common human-AI interaction types (e.g., guiding system, selecting from system outputs, post-editing outputs) in the context of LLM-assisted news headline generation. While LLMs alone can generate satisfactory news headlines, on average, human control is needed to fix undesirable model outputs. Of the interaction methods, guiding and selecting model output added the most benefit with the lowest cost (in time and effort). Further, AI assistance did not harm participants’ perception of control compared to freeform editing.
Anthology ID:
2023.findings-emnlp.217
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3321–3339
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.217
DOI:
10.18653/v1/2023.findings-emnlp.217
Bibkey:
Cite (ACL):
Zijian Ding, Alison Smith-Renner, Wenjuan Zhang, Joel Tetreault, and Alejandro Jaimes. 2023. Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3321–3339, Singapore. Association for Computational Linguistics.
Cite (Informal):
Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation (Ding et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-emnlp.217.pdf