Human Perception in Natural Language Generation

Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim


Abstract
We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.
Anthology ID:
2021.gem-1.2
Volume:
Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | GEM | IJCNLP
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
15–23
Language:
URL:
https://aclanthology.org/2021.gem-1.2
DOI:
10.18653/v1/2021.gem-1.2
Bibkey:
Cite (ACL):
Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, and Malvina Nissim. 2021. Human Perception in Natural Language Generation. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 15–23, Online. Association for Computational Linguistics.
Cite (Informal):
Human Perception in Natural Language Generation (De Mattei et al., GEM 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.gem-1.2.pdf