<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yanaka-mineshima-2022-compositional">
    <titleInfo>
        <title>Compositional Evaluation on Japanese Textual Entailment and Similarity</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hitomi</namePart>
        <namePart type="family">Yanaka</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Koji</namePart>
        <namePart type="family">Mineshima</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Natural Language Inference (NLI) and Semantic Textual Similarity (STS) are widely used benchmark tasks for compositional evaluation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclusively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically different from English and can shed light on the currently controversial behavior of language models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. We also present a stress-test dataset for compositional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on different pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking.</abstract>
    <identifier type="citekey">yanaka-mineshima-2022-compositional</identifier>
    <identifier type="doi">10.1162/tacl_a_00518</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.73</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>1266</start>
            <end>1284</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Compositional Evaluation on Japanese Textual Entailment and Similarity
%A Yanaka, Hitomi
%A Mineshima, Koji
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F yanaka-mineshima-2022-compositional
%X Natural Language Inference (NLI) and Semantic Textual Similarity (STS) are widely used benchmark tasks for compositional evaluation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclusively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically different from English and can shed light on the currently controversial behavior of language models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. We also present a stress-test dataset for compositional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on different pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking.
%R 10.1162/tacl_a_00518
%U https://aclanthology.org/2022.tacl-1.73
%U https://doi.org/10.1162/tacl_a_00518
%P 1266-1284

Markdown (Informal)

[Compositional Evaluation on Japanese Textual Entailment and Similarity](https://aclanthology.org/2022.tacl-1.73) (Yanaka & Mineshima, TACL 2022)

• Compositional Evaluation on Japanese Textual Entailment and Similarity (Yanaka & Mineshima, TACL 2022)

ACL

• Hitomi Yanaka and Koji Mineshima. 2022. Compositional Evaluation on Japanese Textual Entailment and Similarity. Transactions of the Association for Computational Linguistics, 10:1266–1284.