Focused Attention Improves Document-Grounded Generation

Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Additionally, we provide a stronger BART baseline for these tasks. Our proposed techniques outperform existing methods on both automated (at least 48% increase in BLEU-4 points) and human evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in modeling these tasks.

… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.yes, that kind of union and solidarity is always needed in order to achieve anything.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.

Cross-Attention Context
Multi-Headed Attention Pre-Trained Tranformer Decoder Self Attention

Document Headed Attention (DoHA)
Meanwhile, newspapers across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.

Cross-Attention Context
Multi-Headed Attention Pre-Trained Tranformer Decoder Self Attention

Cross-Attention Doc
Yes, but it was dangerous for the white house to ban the post from the whitehouse.… One year later, a security guard discovers a break-in progress at watergate complex.

CONTEXT
It was honorable for the other papers to publish in solidarity with the post and times.That had to prove that the people want to know.7

Wikipedia Update Generation
The year 2000 is sometimes abbreviated as "Y2K" (the "Y" stands for "year", and the "K" stands for "kilo-" which means "thousand") The Y2K conspiracy theory claimed that a secret nuclear attack by the United States on 2 January 2000 was planned to begin World War 2.

Wikipedia Update Generation
Error Class % Reference Generation R

Reference is not grounded 5
This was achieved under dead calm conditions as an additional safety measure, whereas the Wrights flew in a 25 mph+ wind to achieve enough airspeed on their early attempts.
This was verified by a video crew present at the test flight.14

•
generate text given a context and a document • generated text should coherently fit the context and contain new information from the document Yes this is a Christmas classic … Bob: Very well said!Some of the critics were … Alice: Yes I found the first half very entertaining.Bob: I feel you!So hard to believe it was released Yes this is a Christmas classic … Bob: Very well said!Some of the critics were … Alice: Yes I found the first half very entertaining.Bob: I feel you!So hard to believe it was released Yes this is a Christmas classic … Bob: Very well said!Some of the critics were … Alice: Yes I found the first half very entertaining.Bob: I feel you!So hard to believe it was released in 1990!Alice: I like the quote "sadistic festival of adult-bashing".Between's Kevin's parents forgetting him and the burglars this movie doesn't make adult looks too bright.Yes this is a Christmas classic … Bob: Very well said!Some of the critics were … Alice: Yes I found the first half very entertaining.Bob: I feel you!So hard to believe it was released in 1990!Alice: I like the quote "sadistic festival of adult-bashing".Between's Kevin's parents forgetting him and the burglars this movie doesn't make adult looks too bright.to the dialogue context (chat history) 2.contains information from the document • CMU Document Grounded Conversations (CMU_DoG) Dataset • Dataset size of 130k utterances!• Pre-trained encoder-decoder model • provide a stronger baseline for the task Models DOC … Meanwhile, newspapers across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.

•
Pre-trained encoder-decoder model • provide a stronger baseline for the task• Context Driven Representation (CoDR)• build a representation of the content in document that is not present in the context• Document Headed Attention (DoHA)• specific focus on document information across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
across the country pick up the story in solidarity with the post and times.The court rules 6 3 in the newspapers' favor, vindicating graham's decision.Nixon demands that the post should be barred from the white house.
Substantial improvements on automated and human evaluation results on three different datasets • Comprehensive manual analysis • Code and trained models are available at https:// github.com/shrimai/Focused-Attention-Improves-Document-Grounded-Generation