<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bhatia-etal-2026-value">
    <titleInfo>
        <title>Value Drifts: Tracing Value Alignment During LLM Post-Training</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mehar</namePart>
        <namePart type="family">Bhatia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shravan</namePart>
        <namePart type="family">Nayak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gaurav</namePart>
        <namePart type="family">Kamath</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marius</namePart>
        <namePart type="family">Mosbach</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Karolina</namePart>
        <namePart type="family">Stańczak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vered</namePart>
        <namePart type="family">Shwartz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Siva</namePart>
        <namePart type="family">Reddy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model’s post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model’s values, and subsequent preference optimization rarely realigns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.1</abstract>
    <identifier type="citekey">bhatia-etal-2026-value</identifier>
    <identifier type="doi">10.1162/tacl.a.790</identifier>
    <location>
        <url>https://aclanthology.org/2026.tacl-1.92/</url>
    </location>
    <part>
        <date>2026</date>
        <detail type="volume"><number>14</number></detail>
        <extent unit="page">
            <start>2033</start>
            <end>2056</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Value Drifts: Tracing Value Alignment During LLM Post-Training
%A Bhatia, Mehar
%A Nayak, Shravan
%A Kamath, Gaurav
%A Mosbach, Marius
%A Stańczak, Karolina
%A Shwartz, Vered
%A Reddy, Siva
%J Transactions of the Association for Computational Linguistics
%D 2026
%V 14
%I MIT Press
%C Cambridge, MA
%F bhatia-etal-2026-value
%X As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model’s post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model’s values, and subsequent preference optimization rarely realigns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.1
%R 10.1162/tacl.a.790
%U https://aclanthology.org/2026.tacl-1.92/
%U https://doi.org/10.1162/tacl.a.790
%P 2033-2056

Markdown (Informal)

[Value Drifts: Tracing Value Alignment During LLM Post-Training](https://aclanthology.org/2026.tacl-1.92/) (Bhatia et al., TACL 2026)

• Value Drifts: Tracing Value Alignment During LLM Post-Training (Bhatia et al., TACL 2026)

ACL

• Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, and Siva Reddy. 2026. Value Drifts: Tracing Value Alignment During LLM Post-Training. Transactions of the Association for Computational Linguistics, 14:2033–2056.