<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="de-wynter-2025-awes">
    <titleInfo>
        <title>Awes, Laws, and Flaws From Today’s LLM Research</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Adrian</namePart>
        <namePart type="family">de Wynter</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wanxiang</namePart>
            <namePart type="family">Che</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Nabende</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Shutova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohammad</namePart>
            <namePart type="given">Taher</namePart>
            <namePart type="family">Pilehvar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-256-5</identifier>
    </relatedItem>
    <abstract>We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works released between 2020 and 2024 based on criteria typical of what is considered good research (e.g. presence of statistical tests and reproducibility), and cross-validate it with arguments that are at the centre of controversy (e.g., claims of emergent behaviour). We find multiple trends, such as declines in ethics disclaimers, a rise of LLMs as evaluators, and an increase on claims of LLM reasoning abilities without leveraging human evaluation. We note that conference checklists are effective at curtailing some of these issues, but balancing velocity and rigour in research cannot solely rely on these. We tie all these findings to findings from recent meta-reviews and extend recommendations on how to address what does, does not, and should work in LLM research.</abstract>
    <identifier type="citekey">de-wynter-2025-awes</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-acl.664</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-acl.664/</url>
    </location>
    <part>
        <date>2025-07</date>
        <extent unit="page">
            <start>12834</start>
            <end>12854</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Awes, Laws, and Flaws From Today’s LLM Research
%A de Wynter, Adrian
%Y Che, Wanxiang
%Y Nabende, Joyce
%Y Shutova, Ekaterina
%Y Pilehvar, Mohammad Taher
%S Findings of the Association for Computational Linguistics: ACL 2025
%D 2025
%8 July
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-256-5
%F de-wynter-2025-awes
%X We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works released between 2020 and 2024 based on criteria typical of what is considered good research (e.g. presence of statistical tests and reproducibility), and cross-validate it with arguments that are at the centre of controversy (e.g., claims of emergent behaviour). We find multiple trends, such as declines in ethics disclaimers, a rise of LLMs as evaluators, and an increase on claims of LLM reasoning abilities without leveraging human evaluation. We note that conference checklists are effective at curtailing some of these issues, but balancing velocity and rigour in research cannot solely rely on these. We tie all these findings to findings from recent meta-reviews and extend recommendations on how to address what does, does not, and should work in LLM research.
%R 10.18653/v1/2025.findings-acl.664
%U https://aclanthology.org/2025.findings-acl.664/
%U https://doi.org/10.18653/v1/2025.findings-acl.664
%P 12834-12854

Markdown (Informal)

[Awes, Laws, and Flaws From Today’s LLM Research](https://aclanthology.org/2025.findings-acl.664/) (de Wynter, Findings 2025)

• Awes, Laws, and Flaws From Today’s LLM Research (de Wynter, Findings 2025)

ACL

• Adrian de Wynter. 2025. Awes, Laws, and Flaws From Today’s LLM Research. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12834–12854, Vienna, Austria. Association for Computational Linguistics.