<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="litschko-etal-2023-establishing">
    <titleInfo>
        <title>Establishing Trustworthiness: Rethinking Tasks and Model Evaluation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Robert</namePart>
        <namePart type="family">Litschko</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Max</namePart>
        <namePart type="family">Müller-Eberstein</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rob</namePart>
        <namePart type="family">van der Goot</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Leon</namePart>
        <namePart type="family">Weber-Genzel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Barbara</namePart>
        <namePart type="family">Plank</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Traditionally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evaluation protocols. With the advent of large language models (LLMs) the community has witnessed a dramatic shift towards general purpose, task-agnostic approaches powered by generative models. As a consequence, the traditional compartmentalized notion of language tasks is breaking down, followed by an increasing challenge for evaluation and analysis. At the same time, LLMs are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time to rethink what constitutes tasks and model evaluation in NLP, and pursue a more holistic view on language, placing trustworthiness at the center. Towards this goal, we review existing compartmentalized approaches for understanding the origins of a model’s functional capacity, and provide recommendations for more multi-faceted evaluation protocols.</abstract>
    <identifier type="citekey">litschko-etal-2023-establishing</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.14</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.14</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>193</start>
            <end>203</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Establishing Trustworthiness: Rethinking Tasks and Model Evaluation
%A Litschko, Robert
%A Müller-Eberstein, Max
%A van der Goot, Rob
%A Weber-Genzel, Leon
%A Plank, Barbara
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F litschko-etal-2023-establishing
%X Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Traditionally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evaluation protocols. With the advent of large language models (LLMs) the community has witnessed a dramatic shift towards general purpose, task-agnostic approaches powered by generative models. As a consequence, the traditional compartmentalized notion of language tasks is breaking down, followed by an increasing challenge for evaluation and analysis. At the same time, LLMs are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time to rethink what constitutes tasks and model evaluation in NLP, and pursue a more holistic view on language, placing trustworthiness at the center. Towards this goal, we review existing compartmentalized approaches for understanding the origins of a model’s functional capacity, and provide recommendations for more multi-faceted evaluation protocols.
%R 10.18653/v1/2023.emnlp-main.14
%U https://aclanthology.org/2023.emnlp-main.14
%U https://doi.org/10.18653/v1/2023.emnlp-main.14
%P 193-203

Markdown (Informal)

[Establishing Trustworthiness: Rethinking Tasks and Model Evaluation](https://aclanthology.org/2023.emnlp-main.14) (Litschko et al., EMNLP 2023)

• Establishing Trustworthiness: Rethinking Tasks and Model Evaluation (Litschko et al., EMNLP 2023)

ACL

• Robert Litschko, Max Müller-Eberstein, Rob van der Goot, Leon Weber-Genzel, and Barbara Plank. 2023. Establishing Trustworthiness: Rethinking Tasks and Model Evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Singapore. Association for Computational Linguistics.