<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="peskoff-stewart-2023-credible">
    <titleInfo>
        <title>Credible without Credit: Domain Experts Assess Generative Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Denis</namePart>
        <namePart type="family">Peskoff</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Brandon</namePart>
        <namePart type="family">Stewart</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language models have recently broken into the public consciousness with the release of the wildly popular ChatGPT. Commentators have argued that language models could replace search engines, make college essays obsolete, or even write academic research papers. All of these tasks rely on accuracy of specialized information which can be difficult to assess for non-experts. Using 10 domain experts across science and culture, we provide an initial assessment of the coherence, conciseness, accuracy, and sourcing of two language models across 100 expert-written questions. While we find the results are consistently cohesive and concise, we find that they are mixed in their accuracy. These results raise questions of the role language models should play in general-purpose and expert knowledge seeking.</abstract>
    <identifier type="citekey">peskoff-stewart-2023-credible</identifier>
    <identifier type="doi">10.18653/v1/2023.acl-short.37</identifier>
    <location>
        <url>https://aclanthology.org/2023.acl-short.37</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>427</start>
            <end>438</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Credible without Credit: Domain Experts Assess Generative Language Models
%A Peskoff, Denis
%A Stewart, Brandon
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F peskoff-stewart-2023-credible
%X Language models have recently broken into the public consciousness with the release of the wildly popular ChatGPT. Commentators have argued that language models could replace search engines, make college essays obsolete, or even write academic research papers. All of these tasks rely on accuracy of specialized information which can be difficult to assess for non-experts. Using 10 domain experts across science and culture, we provide an initial assessment of the coherence, conciseness, accuracy, and sourcing of two language models across 100 expert-written questions. While we find the results are consistently cohesive and concise, we find that they are mixed in their accuracy. These results raise questions of the role language models should play in general-purpose and expert knowledge seeking.
%R 10.18653/v1/2023.acl-short.37
%U https://aclanthology.org/2023.acl-short.37
%U https://doi.org/10.18653/v1/2023.acl-short.37
%P 427-438

Markdown (Informal)

[Credible without Credit: Domain Experts Assess Generative Language Models](https://aclanthology.org/2023.acl-short.37) (Peskoff & Stewart, ACL 2023)

• Credible without Credit: Domain Experts Assess Generative Language Models (Peskoff & Stewart, ACL 2023)

ACL

• Denis Peskoff and Brandon Stewart. 2023. Credible without Credit: Domain Experts Assess Generative Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 427–438, Toronto, Canada. Association for Computational Linguistics.