<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="robinson-etal-2022-task">
    <titleInfo>
        <title>Task-dependent Optimal Weight Combinations for Static Embeddings</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nathaniel</namePart>
        <namePart type="family">Robinson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nathaniel</namePart>
        <namePart type="family">Carlson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Mortensen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Elizabeth</namePart>
        <namePart type="family">Vargas</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Thomas</namePart>
        <namePart type="family">Fackrell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nancy</namePart>
        <namePart type="family">Fulda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Northern European Journal of Language Technology, Volume 8</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Leon</namePart>
            <namePart type="family">Derczynski</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Northern European Association of Language Technology</publisher>
            <place>
                <placeTerm type="text">Copenhagen, Denmark</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVes default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.</abstract>
    <identifier type="citekey">robinson-etal-2022-task</identifier>
    <identifier type="doi">https://doi.org/10.3384/nejlt.2000-1533.2022.4438</identifier>
    <location>
        <url>https://aclanthology.org/2022.nejlt-1.2</url>
    </location>
    <part>
        <date>2022</date>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Task-dependent Optimal Weight Combinations for Static Embeddings
%A Robinson, Nathaniel
%A Carlson, Nathaniel
%A Mortensen, David
%A Vargas, Elizabeth
%A Fackrell, Thomas
%A Fulda, Nancy
%Y Derczynski, Leon
%S Northern European Journal of Language Technology, Volume 8
%D 2022
%I Northern European Association of Language Technology
%C Copenhagen, Denmark
%F robinson-etal-2022-task
%X A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVes default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.
%R https://doi.org/10.3384/nejlt.2000-1533.2022.4438
%U https://aclanthology.org/2022.nejlt-1.2
%U https://doi.org/https://doi.org/10.3384/nejlt.2000-1533.2022.4438

Markdown (Informal)

[Task-dependent Optimal Weight Combinations for Static Embeddings](https://aclanthology.org/2022.nejlt-1.2) (Robinson et al., NEJLT 2022)

• Task-dependent Optimal Weight Combinations for Static Embeddings (Robinson et al., NEJLT 2022)

ACL

• Nathaniel Robinson, Nathaniel Carlson, David Mortensen, Elizabeth Vargas, Thomas Fackrell, and Nancy Fulda. 2022. Task-dependent Optimal Weight Combinations for Static Embeddings. In Northern European Journal of Language Technology, Volume 8, Copenhagen, Denmark. Northern European Association of Language Technology.