<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="robinson-etal-2022-task">
    <titleInfo>
        <title>Task-dependent Optimal Weight Combinations for Static Embeddings</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nathaniel</namePart>
        <namePart type="family">Robinson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nathaniel</namePart>
        <namePart type="family">Carlson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Mortensen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Elizabeth</namePart>
        <namePart type="family">Vargas</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Thomas</namePart>
        <namePart type="family">Fackrell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nancy</namePart>
        <namePart type="family">Fulda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Northern European Journal of Language Technology</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>Linköping University Electronic Press</publisher>
            <place>
                <placeTerm type="text">Linköping, Sweden</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVes default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.</abstract>
    <identifier type="citekey">robinson-etal-2022-task</identifier>
    <identifier type="doi">10.3384/nejlt.2000-1533.2022.4438</identifier>
    <location>
        <url>https://aclanthology.org/2022.nejlt-1.2/</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>8</number></detail>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Task-dependent Optimal Weight Combinations for Static Embeddings
%A Robinson, Nathaniel
%A Carlson, Nathaniel
%A Mortensen, David
%A Vargas, Elizabeth
%A Fackrell, Thomas
%A Fulda, Nancy
%J Northern European Journal of Language Technology
%D 2022
%V 8
%I Linköping University Electronic Press
%C Linköping, Sweden
%F robinson-etal-2022-task
%X A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVes default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.
%R 10.3384/nejlt.2000-1533.2022.4438
%U https://aclanthology.org/2022.nejlt-1.2/
%U https://doi.org/10.3384/nejlt.2000-1533.2022.4438

Markdown (Informal)

[Task-dependent Optimal Weight Combinations for Static Embeddings](https://aclanthology.org/2022.nejlt-1.2/) (Robinson et al., NEJLT 2022)

• Task-dependent Optimal Weight Combinations for Static Embeddings (Robinson et al., NEJLT 2022)

ACL

• Nathaniel Robinson, Nathaniel Carlson, David Mortensen, Elizabeth Vargas, Thomas Fackrell, and Nancy Fulda. 2022. Task-dependent Optimal Weight Combinations for Static Embeddings. Northern European Journal of Language Technology, 8.