Text-in-Context: Token-Level Error Detection for Table-to-Text Generation

Zdeněk Kasner, Simon Mille, Ondřej Dušek


Abstract
We present our Charles-UPF submission for the Shared Task on Evaluating Accuracy in Generated Texts at INLG 2021. Our system can detect the errors automatically using a combination of a rule-based natural language generation (NLG) system and pretrained language models (LMs). We first utilize a rule-based NLG system to generate sentences with facts that can be derived from the input. For each sentence we evaluate, we select a subset of facts which are relevant by measuring semantic similarity to the sentence in question. Finally, we finetune a pretrained language model on annotated data along with the relevant facts for fine-grained error detection. On the test set, we achieve 69% recall and 75% precision with a model trained on a mixture of human-annotated and synthetic data.
Anthology ID:
2021.inlg-1.25
Volume:
Proceedings of the 14th International Conference on Natural Language Generation
Month:
August
Year:
2021
Address:
Aberdeen, Scotland, UK
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
259–265
Language:
URL:
https://aclanthology.org/2021.inlg-1.25
DOI:
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.inlg-1.25.pdf
Code
 kasnerz/accuracysharedtask_cuni-upf
Data
RotoWire