Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets

Hong-Ren Mao, Hung-Yi Lee


Abstract
Paraphrase generation is an interesting and challenging NLP task which has numerous practical applications. In this paper, we analyze datasets commonly used for paraphrase generation research, and show that simply parroting input sentences surpasses state-of-the-art models in the literature when evaluated on standard metrics. Our findings illustrate that a model could be seemingly adept at generating paraphrases, despite only making trivial changes to the input sentence or even none at all.
Anthology ID:
D19-1611
Volume:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Kentaro Inui, Jing Jiang, Vincent Ng, Xiaojun Wan
Venues:
EMNLP | IJCNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
5960–5968
Language:
URL:
https://aclanthology.org/D19-1611
DOI:
10.18653/v1/D19-1611
Bibkey:
Cite (ACL):
Hong-Ren Mao and Hung-Yi Lee. 2019. Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5960–5968, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets (Mao & Lee, EMNLP-IJCNLP 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-1611.pdf
Attachment:
 D19-1611.Attachment.zip
Data
MS COCO