<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-etal-2021-legoeval">
    <titleInfo>
        <title>LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yu</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Josh</namePart>
        <namePart type="family">Arnold</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Feifan</namePart>
        <namePart type="family">Yan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Weiyan</namePart>
        <namePart type="family">Shi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhou</namePart>
        <namePart type="family">Yu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Heng</namePart>
            <namePart type="family">Ji</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jong</namePart>
            <namePart type="given">C</namePart>
            <namePart type="family">Park</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rui</namePart>
            <namePart type="family">Xia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.</abstract>
    <identifier type="citekey">li-etal-2021-legoeval</identifier>
    <identifier type="doi">10.18653/v1/2021.acl-demo.38</identifier>
    <location>
        <url>https://aclanthology.org/2021.acl-demo.38</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>317</start>
            <end>324</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing
%A Li, Yu
%A Arnold, Josh
%A Yan, Feifan
%A Shi, Weiyan
%A Yu, Zhou
%Y Ji, Heng
%Y Park, Jong C.
%Y Xia, Rui
%S Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations
%D 2021
%8 August
%I Association for Computational Linguistics
%C Online
%F li-etal-2021-legoeval
%X We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.
%R 10.18653/v1/2021.acl-demo.38
%U https://aclanthology.org/2021.acl-demo.38
%U https://doi.org/10.18653/v1/2021.acl-demo.38
%P 317-324

Markdown (Informal)

[LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing](https://aclanthology.org/2021.acl-demo.38) (Li et al., ACL-IJCNLP 2021)

• LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing (Li et al., ACL-IJCNLP 2021)

ACL

• Yu Li, Josh Arnold, Feifan Yan, Weiyan Shi, and Zhou Yu. 2021. LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 317–324, Online. Association for Computational Linguistics.