<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="patil-etal-2021-vyakarana">
    <titleInfo>
        <title>Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rajaswa</namePart>
        <namePart type="family">Patil</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jasleen</namePart>
        <namePart type="family">Dhillon</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Siddhant</namePart>
        <namePart type="family">Mahurkar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Saumitra</namePart>
        <namePart type="family">Kulkarni</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Manav</namePart>
        <namePart type="family">Malhotra</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Veeky</namePart>
        <namePart type="family">Baths</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st Workshop on Multilingual Representation Learning</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Duygu</namePart>
            <namePart type="family">Ataman</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alexandra</namePart>
            <namePart type="family">Birch</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alexis</namePart>
            <namePart type="family">Conneau</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Orhan</namePart>
            <namePart type="family">Firat</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastian</namePart>
            <namePart type="family">Ruder</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gozde</namePart>
            <namePart type="given">Gul</namePart>
            <namePart type="family">Sahin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While there has been significant progress towards developing NLU resources for Indic languages, syntactic evaluation has been relatively less explored. Unlike English, Indic languages have rich morphosyntax, grammatical genders, free linear word-order, and highly inflectional morphology. In this paper, we introduce Vyākarana: a benchmark of Colorless Green sentences in Indic languages for syntactic evaluation of multilingual language models. The benchmark comprises four syntax-related tasks: PoS Tagging, Syntax Tree-depth Prediction, Grammatical Case Marking, and Subject-Verb Agreement. We use the datasets from the evaluation tasks to probe five multilingual language models of varying architectures for syntax in Indic languages. Due to its prevalence, we also include a code-switching setting in our experiments. Our results show that the token-level and sentence-level representations from the Indic language models (IndicBERT and MuRIL) do not capture the syntax in Indic languages as efficiently as the other highly multilingual language models. Further, our layer-wise probing experiments reveal that while mBERT, DistilmBERT, and XLM-R localize the syntax in middle layers, the Indic language models do not show such syntactic localization.</abstract>
    <identifier type="citekey">patil-etal-2021-vyakarana</identifier>
    <identifier type="doi">10.18653/v1/2021.mrl-1.14</identifier>
    <location>
        <url>https://aclanthology.org/2021.mrl-1.14</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>153</start>
            <end>165</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages
%A Patil, Rajaswa
%A Dhillon, Jasleen
%A Mahurkar, Siddhant
%A Kulkarni, Saumitra
%A Malhotra, Manav
%A Baths, Veeky
%Y Ataman, Duygu
%Y Birch, Alexandra
%Y Conneau, Alexis
%Y Firat, Orhan
%Y Ruder, Sebastian
%Y Sahin, Gozde Gul
%S Proceedings of the 1st Workshop on Multilingual Representation Learning
%D 2021
%8 November
%I Association for Computational Linguistics
%C Punta Cana, Dominican Republic
%F patil-etal-2021-vyakarana
%X While there has been significant progress towards developing NLU resources for Indic languages, syntactic evaluation has been relatively less explored. Unlike English, Indic languages have rich morphosyntax, grammatical genders, free linear word-order, and highly inflectional morphology. In this paper, we introduce Vyākarana: a benchmark of Colorless Green sentences in Indic languages for syntactic evaluation of multilingual language models. The benchmark comprises four syntax-related tasks: PoS Tagging, Syntax Tree-depth Prediction, Grammatical Case Marking, and Subject-Verb Agreement. We use the datasets from the evaluation tasks to probe five multilingual language models of varying architectures for syntax in Indic languages. Due to its prevalence, we also include a code-switching setting in our experiments. Our results show that the token-level and sentence-level representations from the Indic language models (IndicBERT and MuRIL) do not capture the syntax in Indic languages as efficiently as the other highly multilingual language models. Further, our layer-wise probing experiments reveal that while mBERT, DistilmBERT, and XLM-R localize the syntax in middle layers, the Indic language models do not show such syntactic localization.
%R 10.18653/v1/2021.mrl-1.14
%U https://aclanthology.org/2021.mrl-1.14
%U https://doi.org/10.18653/v1/2021.mrl-1.14
%P 153-165

Markdown (Informal)

[Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages](https://aclanthology.org/2021.mrl-1.14) (Patil et al., MRL 2021)

• Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages (Patil et al., MRL 2021)

ACL

• Rajaswa Patil, Jasleen Dhillon, Siddhant Mahurkar, Saumitra Kulkarni, Manav Malhotra, and Veeky Baths. 2021. Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 153–165, Punta Cana, Dominican Republic. Association for Computational Linguistics.