<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="basirat-2024-efficient">
    <titleInfo>
        <title>Efficient Structured Prediction with Transformer Encoders</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ali</namePart>
        <namePart type="family">Basirat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Northern European Journal of Language Technology</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>Linköping University Electronic Press</publisher>
            <place>
                <placeTerm type="text">Linköping, Sweden</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Finetuning is a useful method for adapting Transformer-based text encoders to new tasks but can be computationally expensive for structured prediction tasks that require tuning at the token level. Furthermore, finetuning is inherently inefficient in updating all base model parameters, which prevents parameter sharing across tasks. To address these issues, we propose a method for efficient task adaptation of frozen Transformer encoders based on the local contribution of their intermediate layers to token representations. Our adapter uses a novel attention mechanism to aggregate intermediate layers and tailor the resulting representations to a target task. Experiments on several structured prediction tasks demonstrate that our method outperforms previous approaches, retaining over 99% of the finetuning performance at a fraction of the training cost. Our proposed method offers an efficient solution for adapting frozen Transformer encoders to new tasks, improving performance and enabling parameter sharing across different tasks.</abstract>
    <identifier type="citekey">basirat-2024-efficient</identifier>
    <identifier type="doi">10.3384/nejlt.2000-1533.2024.4932</identifier>
    <location>
        <url>https://aclanthology.org/2024.nejlt-1.1/</url>
    </location>
    <part>
        <date>2024-12</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>1</start>
            <end>13</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Efficient Structured Prediction with Transformer Encoders
%A Basirat, Ali
%J Northern European Journal of Language Technology
%D 2024
%8 December
%V 10
%I Linköping University Electronic Press
%C Linköping, Sweden
%F basirat-2024-efficient
%X Finetuning is a useful method for adapting Transformer-based text encoders to new tasks but can be computationally expensive for structured prediction tasks that require tuning at the token level. Furthermore, finetuning is inherently inefficient in updating all base model parameters, which prevents parameter sharing across tasks. To address these issues, we propose a method for efficient task adaptation of frozen Transformer encoders based on the local contribution of their intermediate layers to token representations. Our adapter uses a novel attention mechanism to aggregate intermediate layers and tailor the resulting representations to a target task. Experiments on several structured prediction tasks demonstrate that our method outperforms previous approaches, retaining over 99% of the finetuning performance at a fraction of the training cost. Our proposed method offers an efficient solution for adapting frozen Transformer encoders to new tasks, improving performance and enabling parameter sharing across different tasks.
%R 10.3384/nejlt.2000-1533.2024.4932
%U https://aclanthology.org/2024.nejlt-1.1/
%U https://doi.org/10.3384/nejlt.2000-1533.2024.4932
%P 1-13

Markdown (Informal)

[Efficient Structured Prediction with Transformer Encoders](https://aclanthology.org/2024.nejlt-1.1/) (Basirat, NEJLT 2024)

• Efficient Structured Prediction with Transformer Encoders (Basirat, NEJLT 2024)

ACL

• Ali Basirat. 2024. Efficient Structured Prediction with Transformer Encoders. Northern European Journal of Language Technology, 10:1–13.