<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wu-varshney-2025-transformer">
    <titleInfo>
        <title>Transformer-based Causal Language Models Perform Clustering</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xinbo</namePart>
        <namePart type="family">Wu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lav</namePart>
        <namePart type="given">R</namePart>
        <namePart type="family">Varshney</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: NAACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Luis</namePart>
            <namePart type="family">Chiruzzo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alan</namePart>
            <namePart type="family">Ritter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lu</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Albuquerque, New Mexico</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-195-7</identifier>
    </relatedItem>
    <abstract>Even though large language models (LLMs) have demonstrated remarkable capability in solving various natural language tasks, the capability of an LLM to follow human instructions is still an area of active development. Recent works (Ouyang et al., 2022; Rafailov et al., 2023; Zhang et al., 2023) have shown great improvements in instruction-following capability through additional training for instruction-following tasks. However, the mechanisms responsible for effective instruction-following capabilities remain inadequately understood. Here, we introduce a simplified instruction-following task and use synthetic datasets to analyze a Transformer-based causal language model. Our findings suggest that the model learns task-specific information by clustering data within its hidden space, with this clustering process evolving dynamically during learning. We also demonstrate how this phenomenon assists the model in handling unseen instances, and validate our results in a more realistic setting. We further present applications in pre-training and alignment, inspired by clustering.</abstract>
    <identifier type="citekey">wu-varshney-2025-transformer</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-naacl.296</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-naacl.296/</url>
    </location>
    <part>
        <date>2025-04</date>
        <extent unit="page">
            <start>5347</start>
            <end>5372</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Transformer-based Causal Language Models Perform Clustering
%A Wu, Xinbo
%A Varshney, Lav R.
%Y Chiruzzo, Luis
%Y Ritter, Alan
%Y Wang, Lu
%S Findings of the Association for Computational Linguistics: NAACL 2025
%D 2025
%8 April
%I Association for Computational Linguistics
%C Albuquerque, New Mexico
%@ 979-8-89176-195-7
%F wu-varshney-2025-transformer
%X Even though large language models (LLMs) have demonstrated remarkable capability in solving various natural language tasks, the capability of an LLM to follow human instructions is still an area of active development. Recent works (Ouyang et al., 2022; Rafailov et al., 2023; Zhang et al., 2023) have shown great improvements in instruction-following capability through additional training for instruction-following tasks. However, the mechanisms responsible for effective instruction-following capabilities remain inadequately understood. Here, we introduce a simplified instruction-following task and use synthetic datasets to analyze a Transformer-based causal language model. Our findings suggest that the model learns task-specific information by clustering data within its hidden space, with this clustering process evolving dynamically during learning. We also demonstrate how this phenomenon assists the model in handling unseen instances, and validate our results in a more realistic setting. We further present applications in pre-training and alignment, inspired by clustering.
%R 10.18653/v1/2025.findings-naacl.296
%U https://aclanthology.org/2025.findings-naacl.296/
%U https://doi.org/10.18653/v1/2025.findings-naacl.296
%P 5347-5372

Markdown (Informal)

[Transformer-based Causal Language Models Perform Clustering](https://aclanthology.org/2025.findings-naacl.296/) (Wu & Varshney, Findings 2025)

• Transformer-based Causal Language Models Perform Clustering (Wu & Varshney, Findings 2025)

ACL

• Xinbo Wu and Lav R. Varshney. 2025. Transformer-based Causal Language Models Perform Clustering. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5347–5372, Albuquerque, New Mexico. Association for Computational Linguistics.