<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ulmer-etal-2024-non">
    <titleInfo>
        <title>Non-Exchangeable Conformal Language Generation with Nearest Neighbors</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Dennis</namePart>
        <namePart type="family">Ulmer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chrysoula</namePart>
        <namePart type="family">Zerva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Andre</namePart>
        <namePart type="family">Martins</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Quantifying uncertainty in automatically generated text is important for letting humans check potential hallucinations and making systems more reliable. Conformal prediction is an attractive framework to provide predictions imbued with statistical guarantees, however, its application to text generation is challenging since any i.i.d. assumptions are not realistic. In this paper, we bridge this gap by leveraging recent results on *non-exchangeable* conformal prediction, which still ensures bounds on coverage. The result, *non-exchangeable conformal nucleus sampling*, is a novel extension of the conformal prediction framework to generation based on nearest neighbors. Our method can be used post-hoc for an arbitrary model without extra training and supplies token-level, calibrated prediction sets equipped with statistical guarantees. Experiments in machine translation and language modeling show encouraging results in generation quality. By also producing tighter prediction sets with good coverage, we thus give a more theoretically principled way to perform sampling with conformal guarantees.</abstract>
    <identifier type="citekey">ulmer-etal-2024-non</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-eacl.129</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>1909</start>
            <end>1929</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Non-Exchangeable Conformal Language Generation with Nearest Neighbors
%A Ulmer, Dennis
%A Zerva, Chrysoula
%A Martins, Andre
%Y Graham, Yvette
%Y Purver, Matthew
%S Findings of the Association for Computational Linguistics: EACL 2024
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F ulmer-etal-2024-non
%X Quantifying uncertainty in automatically generated text is important for letting humans check potential hallucinations and making systems more reliable. Conformal prediction is an attractive framework to provide predictions imbued with statistical guarantees, however, its application to text generation is challenging since any i.i.d. assumptions are not realistic. In this paper, we bridge this gap by leveraging recent results on *non-exchangeable* conformal prediction, which still ensures bounds on coverage. The result, *non-exchangeable conformal nucleus sampling*, is a novel extension of the conformal prediction framework to generation based on nearest neighbors. Our method can be used post-hoc for an arbitrary model without extra training and supplies token-level, calibrated prediction sets equipped with statistical guarantees. Experiments in machine translation and language modeling show encouraging results in generation quality. By also producing tighter prediction sets with good coverage, we thus give a more theoretically principled way to perform sampling with conformal guarantees.
%U https://aclanthology.org/2024.findings-eacl.129
%P 1909-1929

Markdown (Informal)

[Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://aclanthology.org/2024.findings-eacl.129) (Ulmer et al., Findings 2024)

• Non-Exchangeable Conformal Language Generation with Nearest Neighbors (Ulmer et al., Findings 2024)

ACL

• Dennis Ulmer, Chrysoula Zerva, and Andre Martins. 2024. Non-Exchangeable Conformal Language Generation with Nearest Neighbors. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1909–1929, St. Julian’s, Malta. Association for Computational Linguistics.