<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wen-yi-mimno-2023-hyperpolyglot">
    <titleInfo>
        <title>Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Andrea</namePart>
        <namePart type="given">W</namePart>
        <namePart type="family">Wen-Yi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Mimno</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Cross-lingual transfer learning is an important property of multilingual large language models (LLMs). But how do LLMs represent relationships between languages? Every language model has an input layer that maps tokens to vectors. This ubiquitous layer of language models is often overlooked. We find that similarities between these input embeddings are highly interpretable and that the geometry of these embeddings differs between model families. In one case (XLM-RoBERTa), embeddings encode language: tokens in different writing systems can be linearly separated with an average of 99.2% accuracy. Another family (mT5) represents cross-lingual semantic similarity: the 50 nearest neighbors for any token represent an average of 7.61 writing systems, and are frequently translations. This result is surprising given that there is no explicit parallel cross-lingual training corpora and no explicit incentive for translations in pre-training objectives. Our research opens the door for investigations in 1) The effect of pre-training and model architectures on representations of languages and 2) The applications of cross-lingual representations embedded in language models.</abstract>
    <identifier type="citekey">wen-yi-mimno-2023-hyperpolyglot</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.71</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.71</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>1124</start>
            <end>1131</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings
%A Wen-Yi, Andrea W.
%A Mimno, David
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F wen-yi-mimno-2023-hyperpolyglot
%X Cross-lingual transfer learning is an important property of multilingual large language models (LLMs). But how do LLMs represent relationships between languages? Every language model has an input layer that maps tokens to vectors. This ubiquitous layer of language models is often overlooked. We find that similarities between these input embeddings are highly interpretable and that the geometry of these embeddings differs between model families. In one case (XLM-RoBERTa), embeddings encode language: tokens in different writing systems can be linearly separated with an average of 99.2% accuracy. Another family (mT5) represents cross-lingual semantic similarity: the 50 nearest neighbors for any token represent an average of 7.61 writing systems, and are frequently translations. This result is surprising given that there is no explicit parallel cross-lingual training corpora and no explicit incentive for translations in pre-training objectives. Our research opens the door for investigations in 1) The effect of pre-training and model architectures on representations of languages and 2) The applications of cross-lingual representations embedded in language models.
%R 10.18653/v1/2023.emnlp-main.71
%U https://aclanthology.org/2023.emnlp-main.71
%U https://doi.org/10.18653/v1/2023.emnlp-main.71
%P 1124-1131

Markdown (Informal)

[Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings](https://aclanthology.org/2023.emnlp-main.71) (Wen-Yi & Mimno, EMNLP 2023)

• Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings (Wen-Yi & Mimno, EMNLP 2023)

ACL

• Andrea W Wen-Yi and David Mimno. 2023. Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1124–1131, Singapore. Association for Computational Linguistics.