<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kovatchev-etal-2019-qualitative">
    <titleInfo>
        <title>A Qualitative Evaluation Framework for Paraphrase Identification</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Venelin</namePart>
        <namePart type="family">Kovatchev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">M</namePart>
        <namePart type="given">Antonia</namePart>
        <namePart type="family">Marti</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maria</namePart>
        <namePart type="family">Salamo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Javier</namePart>
        <namePart type="family">Beltran</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Galia</namePart>
            <namePart type="family">Angelova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd.</publisher>
            <place>
                <placeTerm type="text">Varna, Bulgaria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we present a new approach for the evaluation, error analysis, and interpretation of supervised and unsupervised Paraphrase Identification (PI) systems. Our evaluation framework makes use of a PI corpus annotated with linguistic phenomena to provide a better understanding and interpretation of the performance of various PI systems. Our approach allows for a qualitative evaluation and comparison of the PI models using human interpretable categories. It does not require modification of the training objective of the systems and does not place additional burden on the developers. We replicate several popular supervised and unsupervised PI systems. Using our evaluation framework we show that: 1) Each system performs differently with respect to a set of linguistic phenomena and makes qualitatively different kinds of errors; 2) Some linguistic phenomena are more challenging than others across all systems.</abstract>
    <identifier type="citekey">kovatchev-etal-2019-qualitative</identifier>
    <identifier type="doi">10.26615/978-954-452-056-4_067</identifier>
    <location>
        <url>https://aclanthology.org/R19-1067</url>
    </location>
    <part>
        <date>2019-09</date>
        <extent unit="page">
            <start>568</start>
            <end>577</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Qualitative Evaluation Framework for Paraphrase Identification
%A Kovatchev, Venelin
%A Marti, M. Antonia
%A Salamo, Maria
%A Beltran, Javier
%Y Mitkov, Ruslan
%Y Angelova, Galia
%S Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
%D 2019
%8 September
%I INCOMA Ltd.
%C Varna, Bulgaria
%F kovatchev-etal-2019-qualitative
%X In this paper, we present a new approach for the evaluation, error analysis, and interpretation of supervised and unsupervised Paraphrase Identification (PI) systems. Our evaluation framework makes use of a PI corpus annotated with linguistic phenomena to provide a better understanding and interpretation of the performance of various PI systems. Our approach allows for a qualitative evaluation and comparison of the PI models using human interpretable categories. It does not require modification of the training objective of the systems and does not place additional burden on the developers. We replicate several popular supervised and unsupervised PI systems. Using our evaluation framework we show that: 1) Each system performs differently with respect to a set of linguistic phenomena and makes qualitatively different kinds of errors; 2) Some linguistic phenomena are more challenging than others across all systems.
%R 10.26615/978-954-452-056-4_067
%U https://aclanthology.org/R19-1067
%U https://doi.org/10.26615/978-954-452-056-4_067
%P 568-577

Markdown (Informal)

[A Qualitative Evaluation Framework for Paraphrase Identification](https://aclanthology.org/R19-1067) (Kovatchev et al., RANLP 2019)

• A Qualitative Evaluation Framework for Paraphrase Identification (Kovatchev et al., RANLP 2019)

ACL

• Venelin Kovatchev, M. Antonia Marti, Maria Salamo, and Javier Beltran. 2019. A Qualitative Evaluation Framework for Paraphrase Identification. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 568–577, Varna, Bulgaria. INCOMA Ltd..