Underreporting of errors in NLG output, and what to do about it

Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, Luou Wen


Abstract
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by ‘state-of-the-art’ research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.
Anthology ID:
2021.inlg-1.14
Volume:
Proceedings of the 14th International Conference on Natural Language Generation
Month:
August
Year:
2021
Address:
Aberdeen, Scotland, UK
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
140–153
Language:
URL:
https://aclanthology.org/2021.inlg-1.14
DOI:
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.inlg-1.14.pdf
Supplementary attachment:
 2021.inlg-1.14.Supplementary_Attachment.zip