<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="saxena-etal-2025-end">
    <titleInfo>
        <title>End-to-End Long Document Summarization using Gradient Caching</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rohit</namePart>
        <namePart type="family">Saxena</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hao</namePart>
        <namePart type="family">Tang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Frank</namePart>
        <namePart type="family">Keller</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Training transformer-based encoder-decoder models for long document summarization poses a significant challenge due to the quadratic memory consumption during training. Several approaches have been proposed to extend the input length at test time, but training with these approaches is still difficult, requiring truncation of input documents and causing a mismatch between training and test conditions. In this work, we propose CachED (Gradient Caching for Encoder-Decoder models), an approach that enables end-to-end training of existing transformer-based encoder-decoder models, using the entire document without truncation. Specifically, we apply non-overlapping sliding windows to input documents, followed by fusion in decoder. During backpropagation, the gradients are cached at the decoder and are passed through the encoder in chunks by re-computing the hidden vectors, similar to gradient checkpointing. In the experiments on long document summarization, we extend BART to CachED BART, processing more than 500K tokens during training and achieving superior performance without using any additional parameters.</abstract>
    <identifier type="citekey">saxena-etal-2025-end</identifier>
    <identifier type="doi">10.1162/tacl.a.40</identifier>
    <location>
        <url>https://aclanthology.org/2025.tacl-1.58/</url>
    </location>
    <part>
        <date>2025</date>
        <detail type="volume"><number>13</number></detail>
        <extent unit="page">
            <start>1271</start>
            <end>1297</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T End-to-End Long Document Summarization using Gradient Caching
%A Saxena, Rohit
%A Tang, Hao
%A Keller, Frank
%J Transactions of the Association for Computational Linguistics
%D 2025
%V 13
%I MIT Press
%C Cambridge, MA
%F saxena-etal-2025-end
%X Training transformer-based encoder-decoder models for long document summarization poses a significant challenge due to the quadratic memory consumption during training. Several approaches have been proposed to extend the input length at test time, but training with these approaches is still difficult, requiring truncation of input documents and causing a mismatch between training and test conditions. In this work, we propose CachED (Gradient Caching for Encoder-Decoder models), an approach that enables end-to-end training of existing transformer-based encoder-decoder models, using the entire document without truncation. Specifically, we apply non-overlapping sliding windows to input documents, followed by fusion in decoder. During backpropagation, the gradients are cached at the decoder and are passed through the encoder in chunks by re-computing the hidden vectors, similar to gradient checkpointing. In the experiments on long document summarization, we extend BART to CachED BART, processing more than 500K tokens during training and achieving superior performance without using any additional parameters.
%R 10.1162/tacl.a.40
%U https://aclanthology.org/2025.tacl-1.58/
%U https://doi.org/10.1162/tacl.a.40
%P 1271-1297

Markdown (Informal)

[End-to-End Long Document Summarization using Gradient Caching](https://aclanthology.org/2025.tacl-1.58/) (Saxena et al., TACL 2025)

• End-to-End Long Document Summarization using Gradient Caching (Saxena et al., TACL 2025)

ACL

• Rohit Saxena, Hao Tang, and Frank Keller. 2025. End-to-End Long Document Summarization using Gradient Caching. Transactions of the Association for Computational Linguistics, 13:1271–1297.