@inproceedings{robinson-etal-2022-task,
    title = "Task-dependent Optimal Weight Combinations for Static Embeddings",
    author = "Robinson, Nathaniel  and
      Carlson, Nathaniel  and
      Mortensen, David  and
      Vargas, Elizabeth  and
      Fackrell, Thomas  and
      Fulda, Nancy",
    booktitle = "Northern European Journal of Language Technology, Volume 8",
    year = "2022",
    address = "Copenhagen, Denmark",
    publisher = "Northern European Association of Language Technology",
    url = "https://aclanthology.org/2022.nejlt-1.2",
    doi = "https://doi.org/10.3384/nejlt.2000-1533.2022.4438",
    abstract = "A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0{\%} improvements. Notably, GloVes default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2{\%}.",
}