Human Perception in Natural Language Generation

Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim


Abstract
We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.
Anthology ID:
2021.gem-1.2
Volume:
Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | GEM | IJCNLP
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
15–23
Language:
URL:
https://aclanthology.org/2021.gem-1.2
DOI:
10.18653/v1/2021.gem-1.2
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.gem-1.2.pdf