<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="grail-etal-2021-globalizing">
    <titleInfo>
        <title>Globalizing BERT-based Transformer Architectures for Long Document Summarization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Quentin</namePart>
        <namePart type="family">Grail</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Julien</namePart>
        <namePart type="family">Perez</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Eric</namePart>
        <namePart type="family">Gaussier</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Paola</namePart>
            <namePart type="family">Merlo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jorg</namePart>
            <namePart type="family">Tiedemann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Reut</namePart>
            <namePart type="family">Tsarfaty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Fine-tuning a large language model on downstream tasks has become a commonly adopted process in the Natural Language Processing (NLP) (CITATION). However, such a process, when associated with the current transformer-based (CITATION) architectures, shows several limitations when the target task requires to reason with long documents. In this work, we introduce a novel hierarchical propagation layer that spreads information between multiple transformer windows. We adopt a hierarchical approach where the input is divided in multiple blocks independently processed by the scaled dot-attentions and combined between the successive layers. We validate the effectiveness of our approach on three extractive summarization corpora of long scientific papers and news articles. We compare our approach to standard and pre-trained language-model-based summarizers and report state-of-the-art results for long document summarization and comparable results for smaller document summarization.</abstract>
    <identifier type="citekey">grail-etal-2021-globalizing</identifier>
    <identifier type="doi">10.18653/v1/2021.eacl-main.154</identifier>
    <location>
        <url>https://aclanthology.org/2021.eacl-main.154</url>
    </location>
    <part>
        <date>2021-04</date>
        <extent unit="page">
            <start>1792</start>
            <end>1810</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Globalizing BERT-based Transformer Architectures for Long Document Summarization
%A Grail, Quentin
%A Perez, Julien
%A Gaussier, Eric
%Y Merlo, Paola
%Y Tiedemann, Jorg
%Y Tsarfaty, Reut
%S Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
%D 2021
%8 April
%I Association for Computational Linguistics
%C Online
%F grail-etal-2021-globalizing
%X Fine-tuning a large language model on downstream tasks has become a commonly adopted process in the Natural Language Processing (NLP) (CITATION). However, such a process, when associated with the current transformer-based (CITATION) architectures, shows several limitations when the target task requires to reason with long documents. In this work, we introduce a novel hierarchical propagation layer that spreads information between multiple transformer windows. We adopt a hierarchical approach where the input is divided in multiple blocks independently processed by the scaled dot-attentions and combined between the successive layers. We validate the effectiveness of our approach on three extractive summarization corpora of long scientific papers and news articles. We compare our approach to standard and pre-trained language-model-based summarizers and report state-of-the-art results for long document summarization and comparable results for smaller document summarization.
%R 10.18653/v1/2021.eacl-main.154
%U https://aclanthology.org/2021.eacl-main.154
%U https://doi.org/10.18653/v1/2021.eacl-main.154
%P 1792-1810

Markdown (Informal)

[Globalizing BERT-based Transformer Architectures for Long Document Summarization](https://aclanthology.org/2021.eacl-main.154) (Grail et al., EACL 2021)

• Globalizing BERT-based Transformer Architectures for Long Document Summarization (Grail et al., EACL 2021)

ACL

• Quentin Grail, Julien Perez, and Eric Gaussier. 2021. Globalizing BERT-based Transformer Architectures for Long Document Summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1792–1810, Online. Association for Computational Linguistics.