<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="shah-etal-2024-development">
    <titleInfo>
        <title>Development of Cognitive Intelligence in Pre-trained Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Raj</namePart>
        <namePart type="given">Sanjay</namePart>
        <namePart type="family">Shah</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Khushi</namePart>
        <namePart type="family">Bhardwaj</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sashank</namePart>
        <namePart type="family">Varma</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). The increasing cognitive alignment of these models has made them candidates for cognitive science theories. Prior research into the emergent cognitive abilities of PLMs has been path-independent to model training, i.e. has only looked at the final model weights and not the intermediate steps. However, building plausible models of human cognition using PLMs also requires aligning their performance during training to the developmental trajectories of children’s thinking. Guided by psychometric tests of human intelligence, we choose four task categories to investigate the alignment of ten popular families of PLMs and evaluate each of their available intermediate and final training steps: Numerical ability, Linguistic abilities, Conceptual understanding, and Fluid reasoning. We find a striking regularity: regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development. Before that window, training appears to endow models with the requisite structure to be poised to rapidly learn from experience. After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.</abstract>
    <identifier type="citekey">shah-etal-2024-development</identifier>
    <identifier type="doi">10.18653/v1/2024.emnlp-main.539</identifier>
    <location>
        <url>https://aclanthology.org/2024.emnlp-main.539/</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>9632</start>
            <end>9657</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Development of Cognitive Intelligence in Pre-trained Language Models
%A Shah, Raj Sanjay
%A Bhardwaj, Khushi
%A Varma, Sashank
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F shah-etal-2024-development
%X Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). The increasing cognitive alignment of these models has made them candidates for cognitive science theories. Prior research into the emergent cognitive abilities of PLMs has been path-independent to model training, i.e. has only looked at the final model weights and not the intermediate steps. However, building plausible models of human cognition using PLMs also requires aligning their performance during training to the developmental trajectories of children’s thinking. Guided by psychometric tests of human intelligence, we choose four task categories to investigate the alignment of ten popular families of PLMs and evaluate each of their available intermediate and final training steps: Numerical ability, Linguistic abilities, Conceptual understanding, and Fluid reasoning. We find a striking regularity: regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development. Before that window, training appears to endow models with the requisite structure to be poised to rapidly learn from experience. After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.
%R 10.18653/v1/2024.emnlp-main.539
%U https://aclanthology.org/2024.emnlp-main.539/
%U https://doi.org/10.18653/v1/2024.emnlp-main.539
%P 9632-9657

Markdown (Informal)

[Development of Cognitive Intelligence in Pre-trained Language Models](https://aclanthology.org/2024.emnlp-main.539/) (Shah et al., EMNLP 2024)

• Development of Cognitive Intelligence in Pre-trained Language Models (Shah et al., EMNLP 2024)

ACL

• Raj Sanjay Shah, Khushi Bhardwaj, and Sashank Varma. 2024. Development of Cognitive Intelligence in Pre-trained Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9632–9657, Miami, Florida, USA. Association for Computational Linguistics.