<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="atil-etal-2025-llms">
    <titleInfo>
        <title>Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Berk</namePart>
        <namePart type="family">Atil</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vipul</namePart>
        <namePart type="family">Gupta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sarkar</namePart>
        <namePart type="given">Snigdha</namePart>
        <namePart type="given">Sarathi</namePart>
        <namePart type="family">Das</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rebecca</namePart>
        <namePart type="family">Passonneau</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the The 9th Workshop on Online Abuse and Harms (WOAH)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Agostina</namePart>
            <namePart type="family">Calabrese</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christine</namePart>
            <namePart type="family">de Kock</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Debora</namePart>
            <namePart type="family">Nozza</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Flor</namePart>
            <namePart type="given">Miriam</namePart>
            <namePart type="family">Plaza-del-Arco</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zeerak</namePart>
            <namePart type="family">Talat</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Francielle</namePart>
            <namePart type="family">Vargas</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-105-6</identifier>
    </relatedItem>
    <abstract>Large language models (LLMs) have become ubiquitous, thus it is important to understand their risks and limitations, such as their propensity to generate harmful output. This includes smaller LLMs, which are important for settings with constrained compute resources, such as edge devices. Detection of LLM harm typically requires human annotation, which is expensive to collect. This work studies two questions: How do smaller LLMs rank regarding generation of harmful content? How well can larger LLMs annotate harmfulness? We prompt three small LLMs to elicit harmful content of various types, such as discriminatory language, offensive content, privacy invasion, or negative influence, and collect human rankings of their outputs. Then, we compare harm annotation from three state-of-the-art large LLMs with each other and with humans. We find that the smaller models differ with respect to harmfulness. We also find that large LLMs show low to moderate agreement with humans.</abstract>
    <identifier type="citekey">atil-etal-2025-llms</identifier>
    <location>
        <url>https://aclanthology.org/2025.woah-1.30/</url>
    </location>
    <part>
        <date>2025-08</date>
        <extent unit="page">
            <start>342</start>
            <end>354</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet
%A Atil, Berk
%A Gupta, Vipul
%A Das, Sarkar Snigdha Sarathi
%A Passonneau, Rebecca
%Y Calabrese, Agostina
%Y de Kock, Christine
%Y Nozza, Debora
%Y Plaza-del-Arco, Flor Miriam
%Y Talat, Zeerak
%Y Vargas, Francielle
%S Proceedings of the The 9th Workshop on Online Abuse and Harms (WOAH)
%D 2025
%8 August
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-105-6
%F atil-etal-2025-llms
%X Large language models (LLMs) have become ubiquitous, thus it is important to understand their risks and limitations, such as their propensity to generate harmful output. This includes smaller LLMs, which are important for settings with constrained compute resources, such as edge devices. Detection of LLM harm typically requires human annotation, which is expensive to collect. This work studies two questions: How do smaller LLMs rank regarding generation of harmful content? How well can larger LLMs annotate harmfulness? We prompt three small LLMs to elicit harmful content of various types, such as discriminatory language, offensive content, privacy invasion, or negative influence, and collect human rankings of their outputs. Then, we compare harm annotation from three state-of-the-art large LLMs with each other and with humans. We find that the smaller models differ with respect to harmfulness. We also find that large LLMs show low to moderate agreement with humans.
%U https://aclanthology.org/2025.woah-1.30/
%P 342-354

Markdown (Informal)

[Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet](https://aclanthology.org/2025.woah-1.30/) (Atil et al., WOAH 2025)

• Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet (Atil et al., WOAH 2025)

ACL

• Berk Atil, Vipul Gupta, Sarkar Snigdha Sarathi Das, and Rebecca Passonneau. 2025. Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet. In Proceedings of the The 9th Workshop on Online Abuse and Harms (WOAH), pages 342–354, Vienna, Austria. Association for Computational Linguistics.