<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-huaping-2024-context">
    <titleInfo>
        <title>Context Length Extension via Generalized Extrapolation Scale</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Linhan</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhang</namePart>
        <namePart type="family">Huaping</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand and virtual meeting</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Context length expansion of transformer models is considered a key challenge, especially when handling context beyond the training length during inference stage. In this paper, we propose Geeneralized extrapolatioN scalE (GeNE), a set of parameterized extrapolation functions applied to each layer and attention head to adaptively adjust its extrapolation scales. Experimental results show that GeNE provides a significant improvement on long context language modeling. By randomly scaling the extrapolation ratio during the finetuning, GeNE achieves stable extrapolation on 64k contexts by training on 16k length text. Further, the instruction following Llama2 model based on GeNE achieved competitive results compared with other open-source models of the same parameter scale.</abstract>
    <identifier type="citekey">li-huaping-2024-context</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.249</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>4211</start>
            <end>4218</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Context Length Extension via Generalized Extrapolation Scale
%A Li, Linhan
%A Huaping, Zhang
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand and virtual meeting
%F li-huaping-2024-context
%X Context length expansion of transformer models is considered a key challenge, especially when handling context beyond the training length during inference stage. In this paper, we propose Geeneralized extrapolatioN scalE (GeNE), a set of parameterized extrapolation functions applied to each layer and attention head to adaptively adjust its extrapolation scales. Experimental results show that GeNE provides a significant improvement on long context language modeling. By randomly scaling the extrapolation ratio during the finetuning, GeNE achieves stable extrapolation on 64k contexts by training on 16k length text. Further, the instruction following Llama2 model based on GeNE achieved competitive results compared with other open-source models of the same parameter scale.
%U https://aclanthology.org/2024.findings-acl.249
%P 4211-4218

Markdown (Informal)

[Context Length Extension via Generalized Extrapolation Scale](https://aclanthology.org/2024.findings-acl.249) (Li & Huaping, Findings 2024)

• Context Length Extension via Generalized Extrapolation Scale (Li & Huaping, Findings 2024)

ACL

• Linhan Li and Zhang Huaping. 2024. Context Length Extension via Generalized Extrapolation Scale. In Findings of the Association for Computational Linguistics ACL 2024, pages 4211–4218, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.