Investigating Pretrained Language Models for Graph-to-Text Generation

Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, Iryna Gurevych


Abstract
Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recent pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that approaches based on PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. We report new state-of-the-art BLEU scores of 49.72 on AMR-LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets - a relative improvement of 31.8%, 4.5%, and 42.4%, respectively, with our models generating significantly more fluent texts than human references. In an extensive analysis, we identify possible reasons for the PLMs’ success on graph-to-text tasks. Our findings suggest that the PLMs benefit from similar facts seen during pretraining or fine-tuning, such that they perform well even when the input graph is reduced to a simple bag of node and edge labels.
Anthology ID:
2021.nlp4convai-1.20
Volume:
Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI
Month:
November
Year:
2021
Address:
Online
Venues:
EMNLP | NLP4ConvAI
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
211–227
Language:
URL:
https://aclanthology.org/2021.nlp4convai-1.20
DOI:
10.18653/v1/2021.nlp4convai-1.20
Bibkey:
Cite (ACL):
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating Pretrained Language Models for Graph-to-Text Generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211–227, Online. Association for Computational Linguistics.
Cite (Informal):
Investigating Pretrained Language Models for Graph-to-Text Generation (Ribeiro et al., NLP4ConvAI 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.nlp4convai-1.20.pdf
Code
 UKPLab/plms-graph2text +  additional community code
Data
AGENDADARTDBpediaLDC2017T10Semantic ScholarWebNLG