<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="amouyal-etal-2024-large">
    <titleInfo>
        <title>Large Language Models for Psycholinguistic Plausibility Pretesting</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Samuel</namePart>
        <namePart type="family">Amouyal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Aya</namePart>
        <namePart type="family">Meltzer-Asscher</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jonathan</namePart>
        <namePart type="family">Berant</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In psycholinguistics, the creation of controlled materials is crucial to ensure that research outcomes are solely attributed to the intended manipulations and not influenced by extraneous factors. To achieve this, psycholinguists typically pretest linguistic materials, where a common pretest is to solicit plausibility judgments from human evaluators on specific sentences. In this work, we investigate whether Language Models (LMs) can be used to generate these plausibility judgements. We investigate a wide range of LMs across multiple linguistic structures and evaluate whether their plausibility judgements correlate with human judgements. We find that GPT-4 plausibility judgements highly correlate with human judgements across the structures we examine, whereas other LMs correlate well with humans on commonly used syntactic structures. We then test whether this correlation implies that LMs can be used instead of humans for pretesting. We find that when coarse-grained plausibility judgements are needed, this works well, but when fine-grained judgements are necessary, even GPT-4 does not provide satisfactory discriminative power.</abstract>
    <identifier type="citekey">amouyal-etal-2024-large</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-eacl.12</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>166</start>
            <end>181</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Large Language Models for Psycholinguistic Plausibility Pretesting
%A Amouyal, Samuel
%A Meltzer-Asscher, Aya
%A Berant, Jonathan
%Y Graham, Yvette
%Y Purver, Matthew
%S Findings of the Association for Computational Linguistics: EACL 2024
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F amouyal-etal-2024-large
%X In psycholinguistics, the creation of controlled materials is crucial to ensure that research outcomes are solely attributed to the intended manipulations and not influenced by extraneous factors. To achieve this, psycholinguists typically pretest linguistic materials, where a common pretest is to solicit plausibility judgments from human evaluators on specific sentences. In this work, we investigate whether Language Models (LMs) can be used to generate these plausibility judgements. We investigate a wide range of LMs across multiple linguistic structures and evaluate whether their plausibility judgements correlate with human judgements. We find that GPT-4 plausibility judgements highly correlate with human judgements across the structures we examine, whereas other LMs correlate well with humans on commonly used syntactic structures. We then test whether this correlation implies that LMs can be used instead of humans for pretesting. We find that when coarse-grained plausibility judgements are needed, this works well, but when fine-grained judgements are necessary, even GPT-4 does not provide satisfactory discriminative power.
%U https://aclanthology.org/2024.findings-eacl.12
%P 166-181

Markdown (Informal)

[Large Language Models for Psycholinguistic Plausibility Pretesting](https://aclanthology.org/2024.findings-eacl.12) (Amouyal et al., Findings 2024)

• Large Language Models for Psycholinguistic Plausibility Pretesting (Amouyal et al., Findings 2024)

ACL

• Samuel Amouyal, Aya Meltzer-Asscher, and Jonathan Berant. 2024. Large Language Models for Psycholinguistic Plausibility Pretesting. In Findings of the Association for Computational Linguistics: EACL 2024, pages 166–181, St. Julian’s, Malta. Association for Computational Linguistics.