Knowledge-Enriched Natural Language Generation

Knowledge-enriched text generation poses unique challenges in modeling and learning, driving active research in several core directions, ranging from integrated modeling of neural representations and symbolic information in the sequential/hierarchical/graphical structures, learning without direct supervisions due to the cost of structured annotation, efficient optimization and inference with massive and global constraints, to language grounding on multiple modalities, and generative reasoning with implicit commonsense knowledge and background knowledge. In this tutorial we will present a roadmap to line up the state-of-the-art methods to tackle these challenges on this cutting-edge problem. We will dive deep into various technical components: how to represent knowledge, how to feed knowledge into a generation model, how to evaluate generation results, and what are the remaining challenges?


Introduction
Natural Language Generation (NLG) aims at deliberately constructing a natural language text in order to meet specified communicative goals. NLG has been applied in many real-world applications, including dialogue systems, biography generation, technical paper draft generation, and multimedia news summarization. Neural language models have achieved impressive successes at automatic NLG, especially on creative writing such as story completion and poetry generation. However, in many downstream applications such as news summarization, counter-argument narrative generation, and knowledge base description, the generation process needs to be guided by certain level of knowledge such as sentiment (Hu et al., 2017), topic , and style (Tikhonov et al., 2019).
Knowledge-enriched text generation poses unique challenges in modeling and learning, driving active research in several core directions, ranging from integrated modeling of neural representations and symbolic information in the sequential/hierarchical/graphical structures, learning without direct supervisions due to the cost of structured annotation, efficient optimization and inference with massive and global constraints, to language grounding on multiple modalities, and generative reasoning with implicit commonsense knowledge and background knowledge. In this tutorial we will present a roadmap to line up the state-of-the-art methods to tackle these challenges on this cuttingedge problem. We will dive deep into various technical components (as shown in Figure 1): how to represent knowledge, how to feed knowledge into a generation model, how to evaluate generation results, and what are the remaining challenges? 2 Brief Tutorial Outline

Motivation and Overview [20 mins]
At the beginning of the tutorial we will motivate the task of knowledge-driven NLG by showing a large variety of applications (e.g., KD and KS) in academia and industry which have been mentioned in the Introduction. We will present examples about the shortcomings of pure Seq2Seq or language models as well as the opportunities of using knowledge to enrich the generation. We categorize the input source knowledge and related advanced machine learning technologies in Figure 1. We will present the overview of this tutorial including language models (LMs) and knowledge representation, general learning and generation frameworks, a variety of NLG methods enriched by knowledge sources including semantics and structures, realworld applications, and discussions. (beyond seq2seq and LM) Figure 1: In this tutorial, we will present advanced NLG methods that inject knowledge from a variety of sources.

General Learning and Generation Frameworks [40 mins]
We will present the general methods of knowledgeenriched NLG, which provide the methodological foundations for incorporating different types of knowledge presented in the subsequent parts. Those methods are categorized into three major paradigms which incorporate knowledge through (1) model architectures that facilitate the use of knowledge, such as attention methods, copy/pointer mechanisms, graph neural networks (GNNs), knowledge-enriched embedding, etc; (2) learning frameworks that inject knowledge information into the generation models through training, such as posterior regularization, constraint-driven learning, semantic loss, knowledge-informed weak supervision, etc; (3) inference methods which imposes on the inference process different knowledge constraints to guide decoding, such as lexical constraints, task-specific objectives, global interdependency, etc.

NLG Methods Enhanced by Various Knowledge Sources: Part I [30 mins]
In this part, we present semantic knowledge-driven natural language generation. The semantic knowledge sources mainly contain keywords, topics, linguistic features, and other semantic constraints (e.g., style, emotion, sentiment). We introduce how the knowledge in each source can be encoded and how the represented knowledge can be decoded into natural language of high quality. In this part, we present structured knowledgedriven natural language generation. The struc-tured knowledge sources mainly contain tables, knowledge bases, and knowledge graphs. We introduce how the knowledge in each source can be represented and integrated into generation frameworks. Then, we introduce the methods that (i) find relevant knowledge (e.g., a relational path) from huge knowledge bases and knowledge graphs, and (ii) construct structured knowledge from text, e.g., OpenIE. Lastly, we introduce recent work that integrates multiple types of knowledge ranging from semantic, unstructured, to structured knowledge. We will give a review of the available structured knowledge representation method, most of which focus on the structured tables. Traditionally, researchers tend to linearize the table for the input with the concatenation of type information. With computational advances in recent years, pre-trained language model based approaches for the linearized input have achieved significant success by combining type information as additional position embedding. However, those methods fail to consider the inter-dependency between different entities. We will discuss two major ways to learn those relations: self-attention mechanism and GNNs.

Applications [30 mins]
In this application session, we first review existing potential applications using the knowledgedriven generations. On one hand, the structured knowledge provides additional guidance for the major tasks such as dialogue systems, video captions , and summarizations. On the other hand, researchers have built independent knowledge guided generation tasks, starting from the data-to-text tasks such as Wikibio generation tasks in low-resource and multilingual setting, Webnlg contests, and RO-TOWIRE, to more complex graph-to-text tasks such as AMR-to-text generation, scientific paper generation tasks, and news comment generation. Then, we will cover various post-processing approaches to enhance the quality of generation results for specific applciations, such as coverage mechanism, self-attention mechanism, and tabletext optimal-transport matching loss. Finally we will briefly present how knowledge-enriched NLG is being used in several conversational AI systems including Amazon Alexa. Other commercial applications for NLG include systems that can retrieve and summarize information from a relational database into natural language text such as Salesforce's Einstein and Tableau.

Remaining Challenges and Future Directions [30 mins]
At the end of the tutorial we will discuss the remaining challenges and some of the future directions, including the challenge of capturing the interdependency of knowledge elements to make generated output coherent, knowledge reasoning, representing time and number, duplicate removal, augmenting massive pre-trained language models with external commonsense and background knowledge, and developing effective automatic evaluation metrics, and rigorous and efficient human evaluation procedures. We will provide pointers to resources, including data sets, software and on-line demos.

Diversity Considerations
The topic to be presented is of great interest to diverse group of audience from academics and industry. We will cover a broad diversity of methods and applications in many languages and domains. In particular, enriching modeling and learning with external knowledge, as the core topic in this tutorial, is particularly helpful for low-resource language modeling where no large data are available.
We have a diverse instructor team across multiple institutions (ND, UIUC, UCSD, and Salesforce Inc.) with varying seniority (ranging from junior/senior PhD students to assistant/full professors and senior researchers), two of whom are female researchers. The team has a diverse and broad expertise in natural language processing and generation, machine learning, data mining, and various application domains.

Prerequisites
This tutorial will present basic and advanced methods in NLG systematically to audience. The audi-ence may find different useful content when have different levels of prior knowledge: (with the number of ⭐ for how much a person may feel comfortable and confident with the subject matter) • Familiar with Machine Learning from text, e.g., "Understand classification tasks and classical supervised methods on text data" (⭐); • Familiar with basic natural language processing (NLP) frameworks, e.g., "Had experience with LSTM, Seq2Seq, transformer" (⭐⭐); • Familiar with some data forms of knowledge, e.g., "Had machine learning experience with topic modeling, knowledge bases, knowledge graphs, data tables, etc." (⭐ ⭐ ⭐).

Presenters
Wenhao Yu is a Ph.D. student in the Department of Computer Science and Engineering at the University of Notre Dame. His research lies in controllable knowledge-driven natural language processing, particularly in natural language generation. His research has been published in top-ranked NLP and data mining conferences such as ACL, EMNLP, AAAI, WWW, and CIKM. Additional information is available at https://wyu97.github.io/ Meng Jiang is an assistant professor in the Department of Computer Science and Engineering at the University of Notre Dame. He received his B.E. and Ph.D. in Computer Science from Tsinghua University and was a postdoctoral research associate at the University of Illinois at Urbana-Champaign. His research interests focus on knowledge graph construction and natural language generation for news summarization and forum post generation. The awads he received include Notre Dame Faculty Award in 2019 and Best Paper Awards at ISDSA and KDD-DLG in 2020. Additional information is available at http://www.meng-jiang.com/.
Zhiting Hu is an assistant professor in Halıcıoglu Data Science Institute at UC San Diego. He received his Ph.D. in Machine Learning from Carnegie Mellon University. His research interest lies in the broad area of natural language processing in particular controllable text generation, machine learning to enable training AI agents from all forms of experiences such as structured knowledge, ML systems and applications. His research was recognized with best demo nomination at ACL 2019 and outstanding paper award at ACL 2016. Additional information is available at http://www.cs.cmu.edu/˜zhitingh/.
Qingyun Wang is a Ph.D. student in the Computer Science Department at the University of Illinois at Urbana-Champaign. His research lies in controllable knowledge-driven natural language generation, with a recent focus on the scientific paper generation. He served as a program committee in generation track for multiple conferences including ICML 2020, ACL 2019-2020, ICLR 2021, etc. He previously entered the finalist of the first Alexa Prize competition. Additional information is available at https://eaglew.github.io/ Heng Ji is a professor at Computer Science Department of University of Illinois at Urbana-Champaign, and Amazon Scholar. She has published on Multimedia Multilingual Information Extraction and Knowledge-enriched NLG including technical paper generation, knowledge base description, and knowledge-aware image and video caption generation. The awards she received include "Young Scientist" by World Economic Forum, "AI's 10 to Watch" Award by IEEE Intelligent Systems, NSF CAREER award, and ACL 2020 Best Demo Award. She has served as the Program Committee Co-Chair of many conferences including NAACL-HLT2018, and she is NAACL secretary 2020-2021. Additional information is available at https://blender.cs. illinois.edu/hengji.html.
Nazneen Rajani is a senior research scientist at Salesforce Research. She got her PhD in Computer Science from UT Austin in 2018. Several of her work has been published in ACL, EMNLP, NACCL, and IJCAI including work on generating explanations for commonsense and physical reasoning. Nazneen was one of the finalists for the VentureBeat Transform 2020 women in AI Research. Her work has been covered by several media outlets including Quanta Magazine, Venture-Beat, SiliconAngle, ZDNet. More information on