Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension

Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models’ fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.

[Sect 1] ... the King said to the huntsmen: "Now, come and show me the little house in the wood." And when he got to the door he knocked at it, and cried, "Little sister, let me in!" ...
[Sect 2] Then the door opened, and the King went in, and there stood a maiden more beautiful than any he had seen before. ... the King looked kindly on her, took her by the hand, and said, "Will you go with me to my castle, and be my dear wife?" ...
• Q1:Why did the King go to the little house?
• A: To ask the maiden to marry him.
• A: To see the maiden.
• Q2:What did the King say when he knocked on the door?

253
To ensure the readability of the text, we made a 254 small number of minor revisions to some obviously 255 outdated vocabulary (e.g., changing "ere" to "be-256 fore") and the unconventional use of punctuation 257 (e.g., changing consecutive semi-colons to periods).

258
For each story, we evaluated the reading difficulty  where/when story events take place and typically 286 start with "Where" or "When."

287
• Action questions ask about characters' behav-

288
iors or additional information about that behav-289 ior.

290
• Feeling questions ask about the character's emo- or making inference. In other words, the answer 327 source is "free-form", meaning that the answers 328 can be any free-text, and there is no limit to 329 where the answer comes from.

407
tions each constitutes about 10% of all questions.

408
Setting and prediction questions are about 5% each.

409
Our dataset contains about 75% explicit questions 410 and 25% implicit questions. See Table 4 for details.    to. We leveraged the commonly-used Rouge-L 447 F1 score for the evaluation of QA performances.

448
For each QA instance, we compared the generated 449 answer with each of the two ground-truth answers 450 and took the higher Rouge-L F1 score.

452
Here in Table 5   gives a comprehensive picture of which reading 484 skills the models lack the most.     Table 9. By comparison, the model 578 trained on NarrativeQA tended to ask general ques-

579
Input story section: the wild people who dwell in the south-west are masters of many black arts. they often lure men of the middle kingdom to their country by promising them their daughters in marriage, but their promises are not to be trusted. once there was the son of a poor family, who agreed to labor for three years for one of the wild men in order to become his son-in-law.
Input Answer 1: The son of a poor family.

Ground-truth Question
Who agreed to labor for three years for one of the wild men in order to become his son-in-law?
Outputs BART-NarQA: What was the son of a poor family? BART-FairytaleQA: Who agreed to labor for one of the wild men in order become his son-in law?
Input Answer 2: The wild people.

Ground-truth Question
Who dwellled in the south-west and were masters of many black arts?

Outputs
BART-NarQA: What dwells in the south-west? BART-FairytaleQA: Who dwell in the south-west are masters of many black arts?       Input story section: you see from this that the sparrow was a truthful bird, and the old woman ought to have been willing to forgive her at once when she asked her pardon so nicely. but not so.the old woman had never loved the sparrow, and had often quarreled with her husband for keeping what she called a dirty bird about the house, saying that it only made extra work for her. now she was only too delighted to have some cause of complaint against the pet. she scolded and even cursed the poor little bird for her bad behavior, and not content with using these harsh, unfeeling words, in a fit of rage she seized the sparrow-who all this time had spread out her wings and bowed her head before the old woman, to show how sorry she was-and fetched the scissors and cut off the poor little bird's tongue.
Input Answer: Cut off the poor little bird's tongue.

Ground-truth Question
What did the woman do to punish the bird?
Outputs BART-NarQA: What did the old woman do in her rage?
BART-FairytaleQA: What did the old woman do after she seized her sparrow?
Input story section: "do not be sparing of the silver pieces in your pocket!" she cried after him as he went off.he went to the village, attended to everything, and came back. the woman tore the cloth apart, made a coat of it and put it on. no sooner had they walked a few miles before they could see a red cloud rising up in the south, like a flying bird."that is my mother," said the woman.in a moment the cloud was overhead. then the woman took the black tea-cups and threw them at it. seven she threw and seven fell to earth again. and then they could hear the mother in the cloud weeping and scolding, and thereupon the cloud disappeared.they went on for about four hours. then they heard a sound like the noise of silk being torn, and could see a cloud as black as ink, which was rushing up against the wind."alas, that is my father!" said the woman. "this is a matter of life and death, for he will not let us be! because of my love for you i will now have to disobey the holiest of laws!" Input Answer: Took the black tea-cups and threw them at it.

Ground-truth Question
What did the wife do when she saw her mother?

Outputs
BART-NarQA: What did the woman do to try and kill her father?
BART-FairytaleQA: What did the woman do after she saw her mother? Causal relationship Q: Why did the two mice come tumbling in, squeaking, and laughing? A: They were being chased by the cat.
Outcome resolution Q: What happened to Timmy after he got in the hamper? A: The hamper takes him to the garden.
Feeling Q: How did the princess feel in her new home? A: happy Prediction Q: How will the other animals treat the duckling? A: The other animals will look down on the duckling.
Explicit Q: How did the girl feel when she saw the old woman's teeth? A: terrified Context: ...but she had such great teeth that the girl was terrified...

Q:
What happened when the door of the stove was opened? A: The flames darted out of its mouth. Context: ...when the door of the stove was opened, the flames darted out of its mouth. This is customary with all stoves...

Implicit
Q: What happened when the prince broke open one of the crow's eggs? A1: The prince found a beautiful palace inside. A2: There was a beautiful palace inside. A3: A little palace was inside and it grew until it covered as much ground as seven large barns.
Context: The Swan Maiden lit in a great wide field, and there she told the prince to break open one of the crow's eggs. The prince did as she bade him, and what should he find but the most beautiful little palace, all of pure gold and silver. He set the palace on the ground, and it grew and grew and grew until it covered as much ground as seven large barns.