<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bakarov-etal-2018-limitations">
    <titleInfo>
        <title>The Limitations of Cross-language Word Embeddings Evaluation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Amir</namePart>
        <namePart type="family">Bakarov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Roman</namePart>
        <namePart type="family">Suvorov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ilya</namePart>
        <namePart type="family">Sochenkov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Malvina</namePart>
            <namePart type="family">Nissim</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jonathan</namePart>
            <namePart type="family">Berant</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alessandro</namePart>
            <namePart type="family">Lenci</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">New Orleans, Louisiana</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The aim of this work is to explore the possible limitations of existing methods of cross-language word embeddings evaluation, addressing the lack of correlation between intrinsic and extrinsic cross-language evaluation methods. To prove this hypothesis, we construct English-Russian datasets for extrinsic and intrinsic evaluation tasks and compare performances of 5 different cross-language models on them. The results say that the scores even on different intrinsic benchmarks do not correlate to each other. We can conclude that the use of human references as ground truth for cross-language word embeddings is not proper unless one does not understand how do native speakers process semantics in their cognition.</abstract>
    <identifier type="citekey">bakarov-etal-2018-limitations</identifier>
    <identifier type="doi">10.18653/v1/S18-2010</identifier>
    <location>
        <url>https://aclanthology.org/S18-2010</url>
    </location>
    <part>
        <date>2018-06</date>
        <extent unit="page">
            <start>94</start>
            <end>100</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Limitations of Cross-language Word Embeddings Evaluation
%A Bakarov, Amir
%A Suvorov, Roman
%A Sochenkov, Ilya
%Y Nissim, Malvina
%Y Berant, Jonathan
%Y Lenci, Alessandro
%S Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics
%D 2018
%8 June
%I Association for Computational Linguistics
%C New Orleans, Louisiana
%F bakarov-etal-2018-limitations
%X The aim of this work is to explore the possible limitations of existing methods of cross-language word embeddings evaluation, addressing the lack of correlation between intrinsic and extrinsic cross-language evaluation methods. To prove this hypothesis, we construct English-Russian datasets for extrinsic and intrinsic evaluation tasks and compare performances of 5 different cross-language models on them. The results say that the scores even on different intrinsic benchmarks do not correlate to each other. We can conclude that the use of human references as ground truth for cross-language word embeddings is not proper unless one does not understand how do native speakers process semantics in their cognition.
%R 10.18653/v1/S18-2010
%U https://aclanthology.org/S18-2010
%U https://doi.org/10.18653/v1/S18-2010
%P 94-100

Markdown (Informal)

[The Limitations of Cross-language Word Embeddings Evaluation](https://aclanthology.org/S18-2010) (Bakarov et al., *SEM 2018)

• The Limitations of Cross-language Word Embeddings Evaluation (Bakarov et al., *SEM 2018)

ACL

• Amir Bakarov, Roman Suvorov, and Ilya Sochenkov. 2018. The Limitations of Cross-language Word Embeddings Evaluation. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 94–100, New Orleans, Louisiana. Association for Computational Linguistics.