Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation

Richard Yuanzhe Pang


Abstract
Regarding the problem of automatically generating paraphrases with modified styles or attributes, the difficulty lies in the lack of parallel corpora. Numerous advances have been proposed for the generation. However, significant problems remain with the auto-evaluation of style transfer tasks. Based on the summary of Pang and Gimpel (2018) and Mir et al. (2019), style transfer evaluations rely on three metrics: post-transfer style classification accuracy, content or semantic similarity, and naturalness or fluency. We elucidate the dangerous current state of style transfer auto-evaluation research. Moreover, we propose ways to aggregate the three metrics into one evaluator. This abstract aims to bring researchers to think about the future of style transfer and style transfer evaluation research.
Anthology ID:
D19-5557
Volume:
Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Wei Xu, Alan Ritter, Tim Baldwin, Afshin Rahimi
Venue:
WNUT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
444–445
Language:
URL:
https://aclanthology.org/D19-5557
DOI:
10.18653/v1/D19-5557
Bibkey:
Cite (ACL):
Richard Yuanzhe Pang. 2019. Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 444–445, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation (Pang, WNUT 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-5557.pdf