<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="haduong-etal-2023-risks">
    <titleInfo>
        <title>Risks and NLP Design: A Case Study on Procedural Document QA</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nikita</namePart>
        <namePart type="family">Haduong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alice</namePart>
        <namePart type="family">Gao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Noah</namePart>
        <namePart type="given">A</namePart>
        <namePart type="family">Smith</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users—and concrete strategies to mitigate them—will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in “zero-shot” mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.</abstract>
    <identifier type="citekey">haduong-etal-2023-risks</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.81</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.81</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>1248</start>
            <end>1269</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Risks and NLP Design: A Case Study on Procedural Document QA
%A Haduong, Nikita
%A Gao, Alice
%A Smith, Noah A.
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F haduong-etal-2023-risks
%X As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users—and concrete strategies to mitigate them—will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in “zero-shot” mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.
%R 10.18653/v1/2023.findings-acl.81
%U https://aclanthology.org/2023.findings-acl.81
%U https://doi.org/10.18653/v1/2023.findings-acl.81
%P 1248-1269

Markdown (Informal)

[Risks and NLP Design: A Case Study on Procedural Document QA](https://aclanthology.org/2023.findings-acl.81) (Haduong et al., Findings 2023)

• Risks and NLP Design: A Case Study on Procedural Document QA (Haduong et al., Findings 2023)

ACL

• Nikita Haduong, Alice Gao, and Noah A. Smith. 2023. Risks and NLP Design: A Case Study on Procedural Document QA. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1248–1269, Toronto, Canada. Association for Computational Linguistics.