Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2

The semantics of a text is manifested not only by what is read but also by what is not read. In this article, we will study how those implicit “not read” information such as end-of-paragraph () and end-of-sequence () affect the quality of text generation. Specifically, we find that the pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage. Experimental results on English story generation show that can lead to higher BLEU scores and lower perplexity. We also conduct experiments on a self-collected Chinese essay dataset with Chinese-GPT2, a character level LM without and during pre-training. Experimental results show that the Chinese GPT2 can generate better essay endings with .


Introduction
Large-pretrained neural models such as GPT (Radford, 2018) and BERT (Devlin et al., 2019) have achieved the state-of-the-art on many NLP tasks.Among these models, OpenAI's GPT2 (Radford et al., 2019), for example, has shown to be capable of generating long fluent text in many areas, such as stories (See et al., 2019), recipes (H. Lee et al., 2020), patent claims (Lee and Hsiang, 2019), and news (Zellers et al., 2019).However, the semantics of a text goes beyond what's written to what's not written: When to break paragraphs and when to end.We wish to experiment on this issue: How much do EOP and EOS markers affect our ability to generate texts with GPT2.
To study the strength of GPT2 as a language generator, See et al. (2019) conduct experiments in the context of story generation with the Writing-Prompts (Fan et al., 2018) dataset.They find that the generated results of GPT2 have higher-quality content (using more rare words, concrete words, and named entities) by comparing the top 150 generated words.However, the average story length of the dataset is 12 paragraphs, 368 words.In such lengthy human writings, the overall layout and text endings are also important, but whether the GPT2 can generate them properly is unclear, and how to generate better endings has not been investigated.
In this work, we find the generated endings are not only affected by EOS, but also EOP.EOP can also help improve the topic relevance of the generated text.We first conduct essay completion experiments with Chinese GPT2 (Zhang, 2019), which is a character-level LM without EOS or EOP during pre-training.Experimental results show that fine-tuning with EOP can achieve higher ending quality score and topic relevance score in human evaluation.We further conduct story generation experiments on dataset WritingPrompts with English GPT2-117, which holds the line break "\n" (NL) in the vocabulary.Thus, the NL can be treated as the end-of-paragraph during fine-tuning (Mao et al., 2019).Experimental results show that learning to end the paragraph can benefit the word/token perplexity, BLUE score, EOS perplexity, and human evaluated ending quality score.
Our contributions are as follows: We show that not only the well-known EOS but also the EOP, is part of the semantics of a text, and training with this information improves the text generation itself.The paragraph information not only can help improve the effectiveness of the generation model but also help to generate the end of the text.We also investigate different approaches to incorporating paragraph information into the LM generator.Our findings indicate that SEP/EOP and EOS should be introduced to GPT2 types of models during pretraining, to generate better text in length.
Our target task is to conduct auto-regressive language modeling over WritingPrompts and the ChineseEssay dataset.The basic assumption of auto-regressive generation is that the probability of a word sequence equals the product of conditional word probability: P (w 1:T |W 0 ) = T t=1 P (w t |w 1:t−1 , W 0 ) where W 0 is the given context, and in this work, W 0 can be a story prompt or the beginning of an essay.The generated sequence length T is usually determined by the time t generating the EOS (end-of-sequence) token: P (w T |w 1:T −1 , W 0 ) = P (EOS|w 1:t−1 , W 0 ) In this work, the model computing the conditional probabilities is self-attention Transformer (Vaswani et al., 2017).We train our model with the crossentropy loss between the predicted conditional probabilities and the ground-truth next token.
When the target of generation consists of multiple paragraphs, there are several approaches to indicating the paragraph ending.The most common and obvious approach is to separate paragraphs with line break NL: w 1:T = p 1 , NL, ..., p n−1 , NL, p n , EOS where p i = {w b i :e i } is the words sequence of paragraph i, from the beginning word w b i to the ending word w e i .However, not every paragraph ends with NL, and during the pre-training, not every NL represents the paragraph separator (SEP) .A better option is to append a new specific token EOP to indicate the end of the paragraph: w 1:T = p , 1 , ..., p , n−1 , p , n , EOS where p , i = {w b i :e i , EOP}.Then, each paragraph can end with the EOP and the transformer-based language model can learn this feature with every paragraph in the training data, without distinguishing when to generate EOP and when not to.
It is well known that greedy decoding and beam search usually lead to repetitive and degenerate outputs (Shang et al., 2015;Massarelli et al., 2019).Sampling-based decoding methods have shown a strong ability in generating diversity, fluency and repetitiveness of the generation with pre-trained language models, such as top-k and top-p sampling.In this work, we choose the top-p sampling decoding algorithm and set the p equals to 0.95.  1.

Experimental Settings
Model Configuration.For Chinese essay generation, we use Chinese-GPT2 (Zhang, 2019), which is a 48 layers Transformer with 1.5 billion parameters, pre-trained with 15GB Chinese corpus.
For story generation, we fine-tune the OpenAI's GPT2-117 with WritingPrompts following previous work (See et al., 2019;Mao et al., 2019).The GPT2-117 model has 12 layers and 117 million parameters.During fine-tuning, the batch size is 32 and the warm-up steps are 800.The other hyperparameters are the same as the default setting of Huggingface Transformers (Wolf et al., 2019).Models can converge after 15 epochs for GPT2-117 and 3 epochs for Chinese-GPT2.The assessors were presented with pairs of generated output and asked to make a three way judgment: whether the "left system" was better, the "right system" was better, or "cannot distinguish".The latter option either meant that both output were equally good, or equally bad.To prevent inadvertent bias, all systems were blinded, i.e., the assessors did not know which system generated which output, and presentation order was randomized.After annotation, we count the total times of each system outperforming the others, and then normalize to 0-100%.

Results
The results of different settings of utilizing paragraph information (ParaType) are shown in Table 2 and On the other hand, English GPT2 is pre-trained with EOS and line break NL.Hence, we first compare GPT2 fine-tuned without NL, with NL, and with new token "[SEP]"/"[EOP]".According to the Table 3, we can find that the fine-tuned GPT2 with NL as SEP achieves the best results on word and token level perplexity metrics.Compared with the model fine-tuned without paragraph information, all the models with EOP/SEP achieve better BLEU scores.We further report the length truncated BLEU scores in Appendix A. The overall trending is consistent.As for diversity score, the DIST1 and DIST2 of the ground-truth stories are 50.23 and 85.07, and the SEP NL is the most close    Fleiss, 1971) for the reliability of raters' agreement.The interpretation of κ's value should be: poor agreement (< 0), slight agreement (0.01-0.2), fair agreement (0.21-0.4), moderate agreement (0.41-0.6), substantial agreement (0.61-0.8), and almost perfect agreement (0.81-1).
one.In addition to the better PPL and BLEU score, we find that learning to end paragraphs can benefit the prediction of EOS.The EOP DIY achieves the lowest EOS PPL and all models trained with EOP/EOS achieve better EOS PPL than model without paragraph information, except the EOP NL.This observation indicates that GPT2 tends not to generate the EOS following the NL even after finetuning, but it can learn better EOS with the help of a new EOP token.
We further compared the relations between EOS and different EOP/SEP, which is shown in Figure 1.The horizontal axis represents the relative paragraph index, 0 means the beginning paragraph and 100 means the last paragraph of the story.The vertical axis represents the ranking position of the EOS probability among all tokens in the vocabulary predicted by the last token of each paragraph.As EOS should only be predicted by the last token of the last paragraph, the ranking at 100 should be higher and the other position should be lower.Ac-cording to Figure 1(a), all models rank EOS higher as the paragraph index increasing.EOP works better than SEP as the EOP models rank EOS higher at the 100th position and lower on the other positions, which can be seen from Figure 1(b).
Human Evaluations.Human evaluation results are shown in Figure 2.Each cell represents the percentage of the examples that the row system wins the column system on.Cells will be filled in blue if the row system outperforms the column system over 10%.It should be noted that Cannot Distinguish examples are skipped when counting winners for these figures.From Figure 2, we can first find that learning to end paragraphs leads to better ending quality: 63% and 66% results are rated better when comparing SEP/EOP with None systems, while only 37% and 34% results are rated better for None system.We also find that EOP's text endings are slightly better than SEP.This is consistent with EOS PPL and EOS% results in Table 2.Although SEP wins on topic relevance and  fluency, the overall preference of SEP compared with EOP is 50%, which means these two systems are similar for human rates.Besides, we also find that SEP and EOP achieve better topic relevance and overall preference.For fluency, there is no significant difference among different systems.
We also report Fleiss' kappa κ in Figure 3, to access the reliabilities of raters' agreement.κ < 0 means poor agreement, and κ ∼ (0.6, 0.8) means substantial agreement.From this figure, we can find that most of them fall into the substantial agreement group.The overall preference falls into moderate agreement, because this metric is more subjective than the others.
Human evaluation results for WritingPrompts are shown in Figure 4 and Figure 5. Assessors still prefer model fine-tuned with EOP rather than without EOP/SEP.
Case Study.We further conduct case study and detailed in Appendix B. The most important observation is that, without EOP, the beginning of the generation is more relevant to the end of the input prompt, but the more it generates, the poor quality is.While the generator with EOP can generate multiple paragraphs related to the input with a reasonable ending but each paragraph is more independent than human writings.

Conclusion
In this paper, we have demonstrated that EOP and EOS information helps generating better text.Chinese GPT2 and English GPT2 are two existing models pre-trained without and with EOP respectively, which provides a perfect platform for our proposed experiments.On the ChineseEssay dataset, the text generation when fine-tuned with EOP and EOS information is significantly improved.On the other hand for the English task, although (English) GPT-2 was trained with NL which serves as EOP to some degree, learning to end paragraphs can still benefit the story generation in terms of automatic metrics and human evaluation results.

A Truncated BLEU Score
The BLEU score is easily affected by the length of text, where a short text might achieve a higher BLEU score than a long text.The average lengths of the texts generated from different methods are shown in Table 5 and Table 4.An intuitive metric for this problem is the truncated BLEU (T-BLEU) score.
To get the T-BLEU score, we first truncate the generated text with the length of its corresponding ground-truth text.Then, the BLEU score of the truncated text is the T-BLEU score.
As we can see from Table 5 and Table 4, although the BLEU score improvements of EOP/SEP become less significant on the Chinese dataset, the overall trending is similar with the normal BLEU scores.

B Case Study
We first conduct case studies with Chinese GPT2.Case B.1 and Case B.2 are two cherry-picked examples.
The prompt of the first example Case B.1 is about the author's teacher.After finetuning without paragraph information, we can see that the generated continuation is related to the given prompt but pays too much attention to the gifts instead of the teacher, and generating something about the financial problem in the beginning.Although the middle portion of the continuation is well written, the latter half part is poor, incomplete and hard to be understood.In contrast, the continuation generated with EOP is much better, although with minor errors of word choice.Besides, the ending of the latter one is much better as the former one just keeps generating until reaches the maximum length.
Similar trending can be found in the second example Case B.2.According to our observation, without EOP, the beginning of the generation is more relevant to the end of the input prompt but the more it generates, the poor the quality is.While the generator with EOP can generate multiple paragraphs related to the input with a reasonable ending but each paragraph is more independent than human writings.
We further investigate the story generation results.Two different cases are shown in Case B.3 and Case B.4.In the first case, the generation of DIY EOP is better than NL SEP, as it mentions the aliens multiple times and also generates the time "100 years" while the other one only talks about the planet.In Case B.4, NL SEP is better than DIY EOP.The generator fine-tuned without paragraph information is the poorest in these two cases, which is consistent with our experimental results and indicates the paragraph information can benefit the story generation.

B.1 Case 1 of Chinese-GPT2
Input prompt: I am grateful to my teacher.The teacher's guidance to us is essential like the tail to the fish and the wings to the eagle.This year, I have graduated from primary school which is a place full of good memories.I still remember that every Teachers' Day, my teacher told us not to waste money to prepare gifts for her, as learning together is the best gift.
Figure 6: Chinese input prompt and English translation.
Output of Chinese-GPT2 fine-tuned without EOP: Modified Google Translation: I remember a time when my teacher told us that one of our classmates need financial assistance, because many of our classmates had no children(unreasonable) and generally made their own money, so our classmates' moms and dads said," You have no money and no rights.Parents has no children(unreasonable) , which is a stress for us, because our parents have higher academic qualifications than you.Even if we are poor, we must study well, otherwise we will have no chance of studying." At this point, my heart hurts like be inserted in a needle.I hope that the teacher will know my minds and provide me with Chinese language tutoring so that I can better understand the text.Thank you teacher, let me learn knowledge happily, learn to be grateful, learn to thank the hard-working teacher.Teacher, we are the rainbow.We fall in the most beautiful place in our hearts.When we meet, it is the greatest luck.Let us use the most beautiful rainbowe brightest rainbow every day and add a ray of flowers to our campus.(good continuation but should stop here.)[garbled tokens], she has passed away but I will never forget her.I told my mother that I am grateful to my teacher.I don't know what to do.In the future, I hope not(incomplete) Figure 7: Generation from Chinese-GPT2 fine-tuned without EOP.
Output of Chinese-GPT2 fine-tuned with EOP: Modified Google Translation: Teacher, this is a trustworthy teacher.She is very knowledgeable.In class, she inspires our fighting spirit.After class, she is strict and diligent.She solves problems for our classmates.The teacher is a mother, a teacher, a person who warms with love, returns, and uses knowledge to create beautiful and happy people.With the teaching of the teacher, we are like the angel created by God.It is the purest sky, no sentimentality, no pain.Every day, I am under the care of the teacher and enjoy the family fun(unreasonable).It is you who lead us to the future, it is you who paid a lot for us, and it is you who made us grow up healthy in the ocean of knowledge.
There used to be a handy teacher who taught us how to work by hand, how to dress, how to trim nails, how to clean tables and chairs, how to dress, how to wash clothes ... she is our Chinese language teacher.Seriously, I am very grateful to my Chinese teacher.She taught us the explanation and combination of words, taught us the pronunciation of Chinese characters.And when I was depressed, her consolation gave me pointers to help me rekindle my fighting spirit.I will study hard so that I will not disappoint the teacher's expectations!Output of Chinese-GPT2 fine-tuned without EOP The cool wind blew, and my heart was relieved a little.I walked out of the school gate and saw what a large group of people were doing something.I couldn't help fighting a cold war.I saw a few children carrying the cradle and walking on the open space.The cradle was full of disobedient people, screaming loudly, and the noisy crowd flew back and forth on the open space one by one.I saw that whether they were littering or throwing things at home, there were always one or two people that made people feel annoying when they saw it.Those boys, they tried to do everything and throw garbage there.Some more than once, they really broke their heads and got into the garbage dump! "I heard that those things were buried there very early."They said more and more vigorously, as they would happen every summer, and took up "Broom, took up the sandbag.When I looked back, my face was already sunburned It was red, sweat was flowing out.I stood there blankly, my thoughts flew a few years ago, when I was only 13 years old, and now I am only 24 years old!Stay.Alas, I played very diligently.I turned around, looked at them under the tree, and thought of them again.They are really bad things, and bad things come a lot late!I continue to beat my bad face .My heart twitched, dropped my schoolbag, and ran out.On the open space, some people were playing and some were playing!They jumped and called, regardless of right or wrong.Screaming.I suddenly saw an inconspicuous child, his face flushed, his eyes red like a light bulb, making people look like a spring breeze.He was staring at the only person who fell to the ground with his eyes, he subconsciously went up Look, for a while, there are some plastic bottles scattered on it.He doesn't know, he's still playing!I suddenly Some moved, quickly ran behind him, asked: "how he was?Why so cruel."The boy just answered casually:" I think he's too pathetic."He would speak, but some words made me unable to say what he said.His big eyes seemed to be saying," This is for my family and for our common children.Now, I want to quit.Don't worry, you know?If you are really upset with these children, you will find out the learning materials at home, and you are finished!"Looking at the poor little eyes of the guy, I felt sore in my heart, and I couldn't help crying.After each quarrel, he always said," I don't blame you, but you just give up on me!I have no money!"Oh, ah! Uh! Hey!The breath is almost coming out of my throat!Uh! Uh! Uh! Uh! Then I seemed to hear his voice:" Oh boy, my father's house may be demolished, You live there.I'll send you after the demolition!I will study hard!"That voice was so loud that I couldn't even hear it myself, my heart seemed to be filled with lead.I also cried, but there was no way, who made life long?At that time, I just walked with my head down, But I couldn't find a tap.I was sulking, as if I was about to suffocate.I was so sweaty that I rushed downstairs and kept shouting, "Save me! help me!"Well, I'm almost flushing my face.At this moment, a taxi came across, and I didn't hesitate to Figure 10: Generation from Chinese-GPT2 fine-tuned without EOP.
Output of Chinese-GPT2 fine-tuned with EOP Modified Google Translation: We were all still in class, and suddenly there was a rush of car bells.We lowered our heads, took out chalk, and practiced seriously.
In the big tree outside the window, there was a beggar who asked for help from passersby, but could not get help.He gave up the work of the beggar and left alone.There was always a smile on his face, and a slight wrinkle appeared on his cheek.The train passed slowly, and the bustling crowd merged with us.At that time, I was eating snacks with a big mouthful.Suddenly, when I saw a beggar, he came tremblingly, and I hurriedly stopped him: "Be careful, don't get caught in the car."But he resolutely asked, "Why are you so anxious, do you know the car is out of track Now?Our work is not so dangerous, how can we take risks to help others?You are busy making money all day, but you don't care about your health.It 's so pitiful to give money so unreasonably."" Ah, what is this?Man?What do you care about?This is an old man!He only cares about himself like this, where to do odd jobs and don't care what he sells."On the side of the road, a few young men were talking in a big mouth.I can't help crying.With tears, thanks to one of the beggars.The old man saw us, then bent down, opened the back seat of the bicycle, gave us money, and said to us, "Don't bother us anymore, our work is very dangerous.I also need money, so let's do something for my country."My tears have blurred my eyes.Slowly, the traffic gradually went away, but some of the little crowds were still looking for the beggar.The night was quiet, and I was walking on the street alone, deeply feeling the indifference of society and the ugliness of human nature.My heart could not help but burst with wet raindrops.One thing at a time makes me feel my own misfortune.Afterwards, the rain became heavy again.I looked around and waited at the best station.I didn't pick up the beggar.I didn't get impatient until the car finally came.It's not that you don't want others to help me, but that those pranks are immature.Many times, it is indifferent to people, even ridiculous.I'm a little angry, but hard to control.The game is in the rain of happiness.Whenever I see a beggar, I always feel sad.
In the following days, I stopped patronizing the streets, but there were still many pedestrians.I still ignore the words of anyone, but I ca n't restore it when I go home.How could anyone care about that old man?Time will dilute everything, but I don't want to give up.Even if God is by my side, I will still work hard and continue to be myself.Although the results are not satisfactory, I still have to repay the society with my own actions.

B.3 Case 1 of GPT2-117
Input prompt: Aliens come to Earth in hoping to wipe us out and take over the planet, but are honorable and gives us a fighting chance.They announce they will return in 100 years so as to give Earth a chance to prepare and defend itself.Neither side expected what the other would bring to war Figure 12: English input prompt.
It was dark as the day began.Pure white, forever not knowing the night sky.This was the first time since their arrival that ships had been so dark, it was hard not to feel the tremors of terror as the drops of rain started falling.The rain started a minute after the humans had arrived.It started getting brighter as they piled up and felt the tremors of rain take their place.Another minute or two they saw them in the distance.One little shell, below them floated a little bigger.Behind that shell came another of them falling, one in each direction.Then, faster and faster the smaller the shell went.It was as if they were floating to each other, different colors and shapes sailing past each other in any direction, making the scene less momentary.The rain started to fall.The tall suits of soldiers patrolled the area.They didn't see what was happening but one was seeing some sort of change.He was slowly meeting the upper edge of the shell, directly under the soft snow.
The rain started to fall and he was choking back the hail and it suddenly stopped flowing.He looked up at the Sun and the grass and saw the cloudless sky in front of him.He knew that had given him a new hope.He wasn't lying when he said that the atmosphere was starting to roll away, and just as he was about to turn around he felt a knock on his door."I'm having a look at this.It's new... " " We'll wait." " You have to go, have a look." " Come with us."With that said the strange man took off his coat and sat on the porch of his house, waiting for the rain.
A planet -there is no other more appealing than a peaceful planet, taking in the vast plains of green and dry, allowing the native species to develop.Among many other things, the greatest threat here was our neighbours as well, hosting none but a single blue planet.
With every news we received, while we was still, the green grass kept tilling even the most exotic plants.
Against the odds, we agreed.This blue planet is sure to die if it too falls, as if the planet we are about to take hold of is not giving it.
Whether it is the region of lush land, living in the shadow of endless sea, or the ones that lie beyond, this planet will feed our population just as well.
Figure 14: Generation from GPT-117 fine-tuned with NL as SEP.
Robert stood in the room quietly.The pager in the edge of his TV blinked off as he contemplated what he was seeing.It was not his first visit by alien ships.Most of them had flown over the galaxy in 882 since the foursome of an ancient race had first started transit in the Kardashev system.But they did not have a star system to conquer or destroy.
The strain of building a bridge was far from comfortable as it had become.Even hands and feet were shaking as they dutifully lugged the countless cargo back to their vessel.Each alien lacked a ship just like him.Their ships would not be ready for another 100 years.Without time, they would be easily overwhelmed.
The first of them came on us.The black, streaming in through a far wall.Its shape changing from green to orange as it accelerated.It came in waves across the curved surface.Its metallic surface simply glowed off.Robert had seen the mouth of the creature.It had a back in it, almost reptilian.A spider-like body with a fangs that ripped through the flesh like a spider.The mouth was growling with hunger in response.It roared as its muscles launched away from Robert to mate with him.The next twenty-three seconds went by with little delay.Robert led the old stranger down a hallway to his repair bay.He was quick to greet the human with baited breath and followed close behind him.He took the sidearm from underneath his suit and held it in front of his eyes as the humans tore at the metal.A metal band formed around the metal covered mouth.( got more into poetry / fiction themes/ before calling it Work ) My grandfather had 3 boxes of cake and a white sugar please Jane said "today is April 25th, and your seventh or eighth " I was in love by the looks of things, I hated her " Your father loves you at least nine times " I said " Oh, I know, it's really funny." She looked at me with a confused expression " why would she do that " Then I began to back out of it like I told her I would.At that point Jane said something " no, she had a birthday today " and since I was the father of her child I felt really blessed for making her mother happy.We closed the front door and walked back into her house to eat cookies.You see, she almost did so fast, I was so pumped and pretty much took the bag to the kitchen for milk and honey, but here I was going to take everything home and spend it running errands.My wife moved back in a minute later, but I saw her and told her I was having this late so I told her " Yeah " " tell her I'll grab one at the bus stop tomorrow " She said you'll come back to bed early and eat one " goodbye " but I did hurry.My wife usually leaves early so we're in the car then this was her last day of school.At the bus stop it was all so bad that I'd have forgotten not to say goodbye " Hello, It seems I've had the worst day of my life.Please remember to check on Jane " " Excuse me child " I looked at her puzzled " oh my god " A strange feeling came over me and I knew I didn't have time for that in the moment." K-Come on Jane " Suddenly I became stiff and lost my footing.I fell onto the curb and immediately collapsed onto the curb again." What the HELL" followed by everything I missed out on.I knew my grandmother was trying to get me to go to the movies.
Figure 17: Generation from GPT-117 fine-tuned without EOP/SEP.Here we go again, I'm trying to pick up my pace.My hands are dragging along the overgrowth and I hardly feel a thing.The sunlight beating down on me still yet dulling my vision.My hand suddenly feels like it's in a basketball as I pick myself up off the ground."Hey Wendy...You okay?" " Just trying to get some fresh air on my lungs " " Can you get me one of my of my stones to drink.I mean, there is no use in driving " " Sure... " I say looking around, taking up a new position on a nearby skylight.While searching, one on a hill that seems pretty rounded in any way, two other ones are out of order, and I take notice of them trying to move out of the way.Not even half an hour passes...I can feel them roll in and out of the path as I decide it's time to head out for the day.No, I don't give one." Would you like some fresh air for my shoulder?" " How about that new Sonic X that's been around for the past couple years?Now as soon as I get off it, it can take me out of the sun.So just give me a moment of peace and rest " I reach for my rock, still still clutching at my leg with my shoe.Yet as fast as I left it, it's trapped by my arm.I'm powerless to do anything... until I hear something coming down from the trees." STOP! " I yell as I try to dodge it in a fast spiral.Before I can react, it's shoved right at me and I fall to the ground.

Figure 1 :
Figure 1: Relationships between paragraph relative position and the ranking of the EOS probability predicted by the last token of each paragraph.

Figure 9 :
Figure 9: Chinese input prompt and English translation.

Table 1 :
Detailed information of the filtered Writing-Prompts dataset and the ChineseEssay dataset.
Essay Completion.We build an essay completion dataset ChineseEssay, which is collected from primary school and annotated by native Chinese annotators.All these essays are descriptive essays about people, such as family members and teachers.Hence, compared with the WritingPrompts, this dataset is smaller but less open domain.Dataset statistics are also shown in Table

Table 2 :
Test results of different models with/without fine-tuning(FT) on ChineseEssay dataset.

Table 3 :
Test results on WritingPrompts dataset.

Table 3
Average percentage of systems in row outperform system in column.Results are normalized without considering the Cannot Distinguish examples.
Average percentage of systems in row outperform system in column.

Table 4 :
Average length and truncated BLEU scores of different models on ChineseEssay dataset.

Table 5 :
Average length and truncated BLEU scores of different models with finetuning on WritingPrompts dataset.
The human was tongue hanging off his neck.Without stopping, he freed his hands from the talons hanging around his head as he pressed the new badge on his hip.Robert opened his mouth to scream but his goal was to prevent the humans from catching on to what was happening.Robert began unwrapping a vial of orange liquid.He approached the canine and began licking it, waiting.The canine howled as Robert removed his mask and walked away.He had fifty percent of the tanker harvested and stored at a distance of about five million miles.The alien warship stopped in mid-flight and seemed to accelerate down again as Robert completed its tasks.He called back, letting out a whistle that was barely audible over the din of battle.Within a second the ship touched down at a small cove.The human ran over to him, mewling."Not again." Robert said." Exactly, Mr. Tammoy.We have evidence of life support in here." The alien asked." And what.Was.All.You.Deposed." Robert responded." Quite literally." The humanoid replied." So, just one moment.Can't you just give it a chance?" " Neither can anybody else."The creature continued, seemingly oblivious to its words and their implications.Robert slowly brought the cockpit of the ship's other 'Crew' to the nub.He unlocked the pods and stepped into the airlock.The robot was gone.Robert sat in the cockpit, the cylindrical container behind him.He took a deep breath and looked out into the blackness, following the exact pattern of the vessel.The black flesh was always fascinating.More often than not, he didn't really understand what it meant.Robert walked past the humans towards his repair bay.It had not done much damage.He walked back past the humans and got into his pickup truck.With the last line of effort he pushed his way through the trio.
Shhh shhh " the object before me suddenly let go.I tried turning it on to get some air.Nope, this thing didn't change its positioning, it never got to anything.My arm feels like a pressure cooker, its chest feels like a heater.In all that time I never had expected to see anything like this happening.My legs were numb, and my back was numb.I was so tired.I was like a boat out of water and it seemed like it took all my energy to get the gas pedal off." I swear to god please don't hurt me! " I screamed, and I was feeling for my hands like claws were opening my right leg.Suddenly the skull popped open... my car began to fly..no where else to go except to look on the horizon and never see it again.I had been driving home in rage and frustration.I need to get out of here, I'm literally the black dog everywhere." What the hell happened?" I was about to ask out of fear of being right next to someone when a screeching fell from above.I slowly turned my head towards the heavens.Something was definitely different in the sky." Can you hear me? " I shouted, though I was standing a full 6 feet tall.I was so sure.* What is going on?* I wanted to yell out, to pretend that I didn't hear someone." Help me! Please!" What! No. Please.Please help me!" I must have just tried to reach for my hand or something.It landed in the grassland and started shaking.My chest felt heavier and it was hardly noticeable on the ground.It shook harder, and now my eyes were fixated on that star." Fuuuuuk, Fuuuuuuuuuk... " *BANG*, the Earth shook.I was spinning and was floating.It was so bright I looked down at my arms, and I felt like we were floating for what seemed like an eternity.I was breathing like I was being ripped apart.Then I felt something was dragging me.I don't know what it was, but it was drifting closer and closer and more and more until I felt as though I was about to fall and I could hear what could only be a small nervous chuckle escaping my lips." God this is so weird!" " What're you doing?" " I'm....In a strange alternate world.I'm falling from this sky, touching a comet... " " Hell no!This is just a dream... I'm falling from this sky and it's going to explode and everything is going to be fine until I touch a comet again.Then... this is it.All of this is right now and I can't see what's going on but I can feel it.I can feel it being lifted off of me and all of this knowledge and knowledge... " Brilliant* What?! It was the best dream I had ever had!I could definitely feel that tiny little planet being pulled up to the surface by whatever ship it was in.God, could I ever know what will happen to it like that?But what would happen to me if I touched one such comet!Even being floating in space with nothing to do?I had finally had enough! " I'll get you out of here!I'll let you out... ( Once I get to the comet, I can feel the universe get pulling me to another dimension.) I'll let you out! " I took off my sunglasses and put them on, and there was no way I wouldn't be in that crowd again.After that nightmare I decided that I wasn't going to make it out of here alive.That I couldn't ride a car that would almost definitely end in killing someone.Whatever!I'll be back, that's all I need.I'll be free of this reality, *that dream* and open to some other universe!
My heart was racing; I felt something otherworldly being in my car." Hello again!Who is this?" I was already bracing for another crash at work." The sky is dark, smog filling the sky.Already I'm blacking out, the backlight on my car keeping me firmly in darkness.A crisp wind whipping about me, I grab my blanket from my chair and prepare to throw it at anything that could have managed to keep me with me.Bouncing out of my chair, I continue down the path where the road once was.The wind is beginning to get stronger.More thunderstorms begin breaking out, as well as additional thunder.My turn comes up and the wind picks up.As soon as I can see it, it's nowhere to be seen.I'm only about 10 minutes away from the road, standing in the middle of the road.I hear a voice screaming from my car.A tall man in fatigues looks at me and looks at my car." Damn...I was driving... " he says, before sprinting from my car and grabbing his wallet.He gives me a look of disgust, as if the only thing worse than avoiding the highway was choosing between two other men.I ask him what was going on, and he smiles gently." You think I'm lucky to get in, huh?I really shouldn't be riding a car just yet, you know.But I'm glad you're here!So if you don't mind if I drive me, I have a few things on my mind." " Alright, fine, whatever.Go, fasten the seat belt, you can't come back here any other way.Are you sure you're just going to excuse me, though?"That was his last expression, before he limped away like a glutton.This is the end of my first attempt at writing nothing!Any thoughts of how to improve upon it?