Neural Methodius Revisited: Do Discourse Relations Help with Pre-Trained Models Too?

Aleksandre Maskharashvili, Symon Stevens-Guille, Xintong Li, Michael White


Abstract
Recent developments in natural language generation (NLG) have bolstered arguments in favor of re-introducing explicit coding of discourse relations in the input to neural models. In the Methodius corpus, a meaning representation (MR) is hierarchically structured and includes discourse relations. Meanwhile pre-trained language models have been shown to implicitly encode rich linguistic knowledge which provides an excellent resource for NLG. By virtue of synthesizing these lines of research, we conduct extensive experiments on the benefits of using pre-trained models and discourse relation information in MRs, focusing on the improvement of discourse coherence and correctness. We redesign the Methodius corpus; we also construct another Methodius corpus in which MRs are not hierarchically structured but flat. We report experiments on different versions of the corpora, which probe when, where, and how pre-trained models benefit from MRs with discourse relation information in them. We conclude that discourse relations significantly improve NLG when data is limited.
Anthology ID:
2021.inlg-1.2
Volume:
Proceedings of the 14th International Conference on Natural Language Generation
Month:
August
Year:
2021
Address:
Aberdeen, Scotland, UK
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
12–23
Language:
URL:
https://aclanthology.org/2021.inlg-1.2
DOI:
Bibkey:
Cite (ACL):
Aleksandre Maskharashvili, Symon Stevens-Guille, Xintong Li, and Michael White. 2021. Neural Methodius Revisited: Do Discourse Relations Help with Pre-Trained Models Too?. In Proceedings of the 14th International Conference on Natural Language Generation, pages 12–23, Aberdeen, Scotland, UK. Association for Computational Linguistics.
Cite (Informal):
Neural Methodius Revisited: Do Discourse Relations Help with Pre-Trained Models Too? (Maskharashvili et al., INLG 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.inlg-1.2.pdf
Code
 aleksadre/methodiusneuralinlg2021