<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="han-etal-2020-empirical">
    <titleInfo>
        <title>An empirical analysis of existing systems and datasets toward general simple question answering</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Namgi</namePart>
        <namePart type="family">Han</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Goran</namePart>
        <namePart type="family">Topic</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hiroshi</namePart>
        <namePart type="family">Noji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hiroya</namePart>
        <namePart type="family">Takamura</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yusuke</namePart>
        <namePart type="family">Miyao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Donia</namePart>
            <namePart type="family">Scott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nuria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Chengqing</namePart>
            <namePart type="family">Zong</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Barcelona, Spain (Online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we evaluate the progress of our field toward solving simple factoid questions over a knowledge base, a practically important problem in natural language interface to database. As in other natural language understanding tasks, a common practice for this task is to train and evaluate a model on a single dataset, and recent studies suggest that SimpleQuestions, the most popular and largest dataset, is nearly solved under this setting. However, this common setting does not evaluate the robustness of the systems outside of the distribution of the used training data. We rigorously evaluate such robustness of existing systems using different datasets. Our analysis, including shifting of training and test datasets and training on a union of the datasets, suggests that our progress in solving SimpleQuestions dataset does not indicate the success of more general simple question answering. We discuss a possible future direction toward this goal.</abstract>
    <identifier type="citekey">han-etal-2020-empirical</identifier>
    <identifier type="doi">10.18653/v1/2020.coling-main.465</identifier>
    <location>
        <url>https://aclanthology.org/2020.coling-main.465</url>
    </location>
    <part>
        <date>2020-12</date>
        <extent unit="page">
            <start>5321</start>
            <end>5334</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T An empirical analysis of existing systems and datasets toward general simple question answering
%A Han, Namgi
%A Topic, Goran
%A Noji, Hiroshi
%A Takamura, Hiroya
%A Miyao, Yusuke
%Y Scott, Donia
%Y Bel, Nuria
%Y Zong, Chengqing
%S Proceedings of the 28th International Conference on Computational Linguistics
%D 2020
%8 December
%I International Committee on Computational Linguistics
%C Barcelona, Spain (Online)
%F han-etal-2020-empirical
%X In this paper, we evaluate the progress of our field toward solving simple factoid questions over a knowledge base, a practically important problem in natural language interface to database. As in other natural language understanding tasks, a common practice for this task is to train and evaluate a model on a single dataset, and recent studies suggest that SimpleQuestions, the most popular and largest dataset, is nearly solved under this setting. However, this common setting does not evaluate the robustness of the systems outside of the distribution of the used training data. We rigorously evaluate such robustness of existing systems using different datasets. Our analysis, including shifting of training and test datasets and training on a union of the datasets, suggests that our progress in solving SimpleQuestions dataset does not indicate the success of more general simple question answering. We discuss a possible future direction toward this goal.
%R 10.18653/v1/2020.coling-main.465
%U https://aclanthology.org/2020.coling-main.465
%U https://doi.org/10.18653/v1/2020.coling-main.465
%P 5321-5334

Markdown (Informal)

[An empirical analysis of existing systems and datasets toward general simple question answering](https://aclanthology.org/2020.coling-main.465) (Han et al., COLING 2020)

• An empirical analysis of existing systems and datasets toward general simple question answering (Han et al., COLING 2020)

ACL

• Namgi Han, Goran Topic, Hiroshi Noji, Hiroya Takamura, and Yusuke Miyao. 2020. An empirical analysis of existing systems and datasets toward general simple question answering. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5321–5334, Barcelona, Spain (Online). International Committee on Computational Linguistics.