GRUEN for Evaluating Linguistic Quality of Generated Text

Wanzheng Zhu, Suma Bhat


Abstract
Automatic evaluation metrics are indispensable for evaluating generated text. To date, these metrics have focused almost exclusively on the content selection aspect of the system output, ignoring the linguistic quality aspect altogether. We bridge this gap by proposing GRUEN for evaluating Grammaticality, non-Redundancy, focUs, structure and coherENce of generated text. GRUEN utilizes a BERT-based model and a class of syntactic, semantic, and contextual features to examine the system output. Unlike most existing evaluation metrics which require human references as an input, GRUEN is reference-less and requires only the system output. Besides, it has the advantage of being unsupervised, deterministic, and adaptable to various tasks. Experiments on seven datasets over four language generation tasks show that the proposed metric correlates highly with human judgments.
Anthology ID:
2020.findings-emnlp.9
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
94–108
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.9
DOI:
10.18653/v1/2020.findings-emnlp.9
Bibkey:
Cite (ACL):
Wanzheng Zhu and Suma Bhat. 2020. GRUEN for Evaluating Linguistic Quality of Generated Text. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 94–108, Online. Association for Computational Linguistics.
Cite (Informal):
GRUEN for Evaluating Linguistic Quality of Generated Text (Zhu & Bhat, Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.9.pdf
Video:
 https://slideslive.com/38940645
Code
 WanzhengZhu/GRUEN +  additional community code
Data
CNN/Daily MailCoLA