<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="deyoung-etal-2024-multi">
    <titleInfo>
        <title>Do Multi-Document Summarization Models Synthesize?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jay</namePart>
        <namePart type="family">DeYoung</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Stephanie</namePart>
        <namePart type="given">C</namePart>
        <namePart type="family">Martinez</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Iain</namePart>
        <namePart type="given">J</namePart>
        <namePart type="family">Marshall</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Byron</namePart>
        <namePart type="given">C</namePart>
        <namePart type="family">Wallace</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately synthesize inputs with respect to a key aspect, e.g., a synopsis of film reviews written about a particular movie should reflect the average critic consensus. As a more consequential example, narrative summaries that accompany biomedical systematic reviews of clinical trial results should accurately summarize the potentially conflicting results from individual trials. In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this sort of synthesis? We run experiments over opinion and evidence synthesis datasets using a suite of summarization models, from fine-tuned transformers to GPT-4. We find that existing models partially perform synthesis, but imperfectly: Even the best performing models are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., ratio of positive to negative reviews). We propose a simple, general, effective method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate.</abstract>
    <identifier type="citekey">deyoung-etal-2024-multi</identifier>
    <identifier type="doi">10.1162/tacl_a_00687</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.58</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>1043</start>
            <end>1062</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Do Multi-Document Summarization Models Synthesize?
%A DeYoung, Jay
%A Martinez, Stephanie C.
%A Marshall, Iain J.
%A Wallace, Byron C.
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F deyoung-etal-2024-multi
%X Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately synthesize inputs with respect to a key aspect, e.g., a synopsis of film reviews written about a particular movie should reflect the average critic consensus. As a more consequential example, narrative summaries that accompany biomedical systematic reviews of clinical trial results should accurately summarize the potentially conflicting results from individual trials. In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this sort of synthesis? We run experiments over opinion and evidence synthesis datasets using a suite of summarization models, from fine-tuned transformers to GPT-4. We find that existing models partially perform synthesis, but imperfectly: Even the best performing models are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., ratio of positive to negative reviews). We propose a simple, general, effective method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate.
%R 10.1162/tacl_a_00687
%U https://aclanthology.org/2024.tacl-1.58
%U https://doi.org/10.1162/tacl_a_00687
%P 1043-1062

Markdown (Informal)

[Do Multi-Document Summarization Models Synthesize?](https://aclanthology.org/2024.tacl-1.58) (DeYoung et al., TACL 2024)

• Do Multi-Document Summarization Models Synthesize? (DeYoung et al., TACL 2024)

ACL

• Jay DeYoung, Stephanie C. Martinez, Iain J. Marshall, and Byron C. Wallace. 2024. Do Multi-Document Summarization Models Synthesize?. Transactions of the Association for Computational Linguistics, 12:1043–1062.