<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mori-etal-2019-toward">
    <titleInfo>
        <title>Toward a Better Story End: Collecting Human Evaluation with Reasons</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yusuke</namePart>
        <namePart type="family">Mori</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hiroaki</namePart>
        <namePart type="family">Yamane</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yusuke</namePart>
        <namePart type="family">Mukuta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tatsuya</namePart>
        <namePart type="family">Harada</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-oct–nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 12th International Conference on Natural Language Generation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kees</namePart>
            <namePart type="family">van Deemter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Chenghua</namePart>
            <namePart type="family">Lin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hiroya</namePart>
            <namePart type="family">Takamura</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Tokyo, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Creativity is an essential element of human nature used for many activities, such as telling a story. Based on human creativity, researchers have attempted to teach a computer to generate stories automatically or support this creative process. In this study, we undertake the task of story ending generation. This is a relatively new task, in which the last sentence of a given incomplete story is automatically generated. This is challenging because, in order to predict an appropriate ending, the generation method should comprehend the context of events. Despite the importance of this task, no clear evaluation metric has been established thus far; hence, it has remained an open problem. Therefore, we study the various elements involved in evaluating an automatic method for generating story endings. First, we introduce a baseline hierarchical sequence-to-sequence method for story ending generation. Then, we conduct a pairwise comparison against human-written endings, in which annotators choose the preferable ending. In addition to a quantitative evaluation, we conduct a qualitative evaluation by asking annotators to specify the reason for their choice. From the collected reasons, we discuss what elements the evaluation should focus on, to thereby propose effective metrics for the task.</abstract>
    <identifier type="citekey">mori-etal-2019-toward</identifier>
    <identifier type="doi">10.18653/v1/W19-8646</identifier>
    <location>
        <url>https://aclanthology.org/W19-8646</url>
    </location>
    <part>
        <date>2019-oct–nov</date>
        <extent unit="page">
            <start>383</start>
            <end>390</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Toward a Better Story End: Collecting Human Evaluation with Reasons
%A Mori, Yusuke
%A Yamane, Hiroaki
%A Mukuta, Yusuke
%A Harada, Tatsuya
%Y van Deemter, Kees
%Y Lin, Chenghua
%Y Takamura, Hiroya
%S Proceedings of the 12th International Conference on Natural Language Generation
%D 2019
%8 oct–nov
%I Association for Computational Linguistics
%C Tokyo, Japan
%F mori-etal-2019-toward
%X Creativity is an essential element of human nature used for many activities, such as telling a story. Based on human creativity, researchers have attempted to teach a computer to generate stories automatically or support this creative process. In this study, we undertake the task of story ending generation. This is a relatively new task, in which the last sentence of a given incomplete story is automatically generated. This is challenging because, in order to predict an appropriate ending, the generation method should comprehend the context of events. Despite the importance of this task, no clear evaluation metric has been established thus far; hence, it has remained an open problem. Therefore, we study the various elements involved in evaluating an automatic method for generating story endings. First, we introduce a baseline hierarchical sequence-to-sequence method for story ending generation. Then, we conduct a pairwise comparison against human-written endings, in which annotators choose the preferable ending. In addition to a quantitative evaluation, we conduct a qualitative evaluation by asking annotators to specify the reason for their choice. From the collected reasons, we discuss what elements the evaluation should focus on, to thereby propose effective metrics for the task.
%R 10.18653/v1/W19-8646
%U https://aclanthology.org/W19-8646
%U https://doi.org/10.18653/v1/W19-8646
%P 383-390

Markdown (Informal)

[Toward a Better Story End: Collecting Human Evaluation with Reasons](https://aclanthology.org/W19-8646) (Mori et al., INLG 2019)

• Toward a Better Story End: Collecting Human Evaluation with Reasons (Mori et al., INLG 2019)

ACL

• Yusuke Mori, Hiroaki Yamane, Yusuke Mukuta, and Tatsuya Harada. 2019. Toward a Better Story End: Collecting Human Evaluation with Reasons. In Proceedings of the 12th International Conference on Natural Language Generation, pages 383–390, Tokyo, Japan. Association for Computational Linguistics.