<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="pimentel-etal-2026-straightforward">
    <titleInfo>
        <title>A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">João</namePart>
        <namePart type="family">Pimentel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Joana</namePart>
        <namePart type="family">Amorim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Frank</namePart>
        <namePart type="family">Rudzicz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Even though Transformers are the standard to extract information from text, they have considerable limitations that make them sub-optimal for certain tasks. Graph neural networks (GNNs) are an alternative that mitigate some of these limitations, with recent work focusing on combining information from pre-trained language models (PLMs) with graph structures. However, most existing methods use fixed-size sliding windows to construct graphs, ignoring long-distance relationships between words. We propose a memory-efficient method to construct token graphs that exploits semantic and relational information inside Transformers via their attention coefficients, mitigating the limitations of using sliding windows. Additionally, each graph can fully encode documents longer than the Transformer’s context window via a chunk-and-stride mechanism, while also reducing memory usage. Our method surpasses the performance of sliding windows when both approaches are compared under the same model architecture, particularly in longer documents, and reaches statistically identical performance in shorter ones. It also surpasses GNN-based state-of-the-art techniques. When compared with fine-tuning, our method reaches marginally lower performance in documents that fit inside the pre-trained model’s context window. However, our method surpasses fine-tuning in documents that do not fit inside these windows, which comes as a result of fully encoding each document using our chunk-and-stride mechanism.</abstract>
    <identifier type="citekey">pimentel-etal-2026-straightforward</identifier>
    <identifier type="doi">10.1162/coli.a.618</identifier>
    <location>
        <url>https://aclanthology.org/2026.cl-3.7/</url>
    </location>
    <part>
        <date>2026-09</date>
        <detail type="volume"><number>52</number></detail>
        <detail type="issue"><number>3</number></detail>
        <extent unit="page">
            <start>1157</start>
            <end>1206</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers
%A Pimentel, João
%A Amorim, Joana
%A Rudzicz, Frank
%J Computational Linguistics
%D 2026
%8 September
%V 52
%N 3
%I MIT Press
%C Cambridge, MA
%F pimentel-etal-2026-straightforward
%X Even though Transformers are the standard to extract information from text, they have considerable limitations that make them sub-optimal for certain tasks. Graph neural networks (GNNs) are an alternative that mitigate some of these limitations, with recent work focusing on combining information from pre-trained language models (PLMs) with graph structures. However, most existing methods use fixed-size sliding windows to construct graphs, ignoring long-distance relationships between words. We propose a memory-efficient method to construct token graphs that exploits semantic and relational information inside Transformers via their attention coefficients, mitigating the limitations of using sliding windows. Additionally, each graph can fully encode documents longer than the Transformer’s context window via a chunk-and-stride mechanism, while also reducing memory usage. Our method surpasses the performance of sliding windows when both approaches are compared under the same model architecture, particularly in longer documents, and reaches statistically identical performance in shorter ones. It also surpasses GNN-based state-of-the-art techniques. When compared with fine-tuning, our method reaches marginally lower performance in documents that fit inside the pre-trained model’s context window. However, our method surpasses fine-tuning in documents that do not fit inside these windows, which comes as a result of fully encoding each document using our chunk-and-stride mechanism.
%R 10.1162/coli.a.618
%U https://aclanthology.org/2026.cl-3.7/
%U https://doi.org/10.1162/coli.a.618
%P 1157-1206

Markdown (Informal)

[A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers](https://aclanthology.org/2026.cl-3.7/) (Pimentel et al., CL 2026)

• A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers (Pimentel et al., CL 2026)

ACL

• João Pimentel, Joana Amorim, and Frank Rudzicz. 2026. A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers. Computational Linguistics, 52(3):1157–1206.