CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, Minlie Huang


Abstract
Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judgments, whereas supervised ones may overfit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtaining better generalization of evaluating generated texts from different models and with different qualities.
Anthology ID:
2022.acl-long.164
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2306–2319
Language:
URL:
https://aclanthology.org/2022.acl-long.164
DOI:
10.18653/v1/2022.acl-long.164
Bibkey:
Cite (ACL):
Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022. CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2306–2319, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (Ke et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.164.pdf
Software:
 2022.acl-long.164.software.zip
Code
 thu-coai/ctrleval