<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wilson-etal-2023-abstract">
    <titleInfo>
        <title>How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Wilson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jackson</namePart>
        <namePart type="family">Petty</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Robert</namePart>
        <namePart type="family">Frank</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.1</abstract>
    <identifier type="citekey">wilson-etal-2023-abstract</identifier>
    <identifier type="doi">10.1162/tacl_a_00608</identifier>
    <location>
        <url>https://aclanthology.org/2023.tacl-1.78</url>
    </location>
    <part>
        <date>2023</date>
        <detail type="volume"><number>11</number></detail>
        <extent unit="page">
            <start>1377</start>
            <end>1395</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
%A Wilson, Michael
%A Petty, Jackson
%A Frank, Robert
%J Transactions of the Association for Computational Linguistics
%D 2023
%V 11
%I MIT Press
%C Cambridge, MA
%F wilson-etal-2023-abstract
%X Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.1
%R 10.1162/tacl_a_00608
%U https://aclanthology.org/2023.tacl-1.78
%U https://doi.org/10.1162/tacl_a_00608
%P 1377-1395

Markdown (Informal)

[How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure](https://aclanthology.org/2023.tacl-1.78) (Wilson et al., TACL 2023)

• How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure (Wilson et al., TACL 2023)

ACL

• Michael Wilson, Jackson Petty, and Robert Frank. 2023. How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure. Transactions of the Association for Computational Linguistics, 11:1377–1395.