<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhao-etal-2026-validity">
    <titleInfo>
        <title>Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Qinlin</namePart>
        <namePart type="family">Zhao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xiaoyuan</namePart>
        <namePart type="family">Yi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jing</namePart>
        <namePart type="family">Yao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Muhua</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Defu</namePart>
        <namePart type="family">Lian</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xing</namePart>
        <namePart type="family">Xie</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Measuring the value orientations of LLMs is helpful for diagnosing model risk and understanding their misalignment behavior. However, existing value benchmarks are often AI-generated and lack validity verification, leading to noisy and unreliable results. Inspired by Psychometrics, we propose a novel Subset Item Response Theory (S-IRT) model for assessing LLMs’ values under any value theories. S-IRT directly models the subset of values jointly expressed in each LLM response, rather than reducing responses to binary outcomes as in traditional IRT-based ability assessment. Meanwhile, it jointly estimates the LLM’s latent values and captures the relative priorities among them. Based on it, we further design a tailored information reward that estimates the information gain of each item in reducing uncertainty about the evaluated model’s value, and thus helps select a small set of high-quality items from existing benchmarks. Extensive experiments on 23 popular LLMs and three value systems (21 value dimensions) demonstrate S-IRT can be applied to multiple value systems simultaneously and improve evaluation reliability and validity. The information reward also helps reduce the number of questions to as few as 200 while preserving comparable evaluation effectiveness.</abstract>
    <identifier type="citekey">zhao-etal-2026-validity</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.2578/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>58592</start>
            <end>58615</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory
%A Zhao, Qinlin
%A Yi, Xiaoyuan
%A Yao, Jing
%A Huang, Muhua
%A Lian, Defu
%A Xie, Xing
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F zhao-etal-2026-validity
%X Measuring the value orientations of LLMs is helpful for diagnosing model risk and understanding their misalignment behavior. However, existing value benchmarks are often AI-generated and lack validity verification, leading to noisy and unreliable results. Inspired by Psychometrics, we propose a novel Subset Item Response Theory (S-IRT) model for assessing LLMs’ values under any value theories. S-IRT directly models the subset of values jointly expressed in each LLM response, rather than reducing responses to binary outcomes as in traditional IRT-based ability assessment. Meanwhile, it jointly estimates the LLM’s latent values and captures the relative priorities among them. Based on it, we further design a tailored information reward that estimates the information gain of each item in reducing uncertainty about the evaluated model’s value, and thus helps select a small set of high-quality items from existing benchmarks. Extensive experiments on 23 popular LLMs and three value systems (21 value dimensions) demonstrate S-IRT can be applied to multiple value systems simultaneously and improve evaluation reliability and validity. The information reward also helps reduce the number of questions to as few as 200 while preserving comparable evaluation effectiveness.
%U https://aclanthology.org/2026.emnlp-main.2578/
%P 58592-58615

Markdown (Informal)

[Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory](https://aclanthology.org/2026.emnlp-main.2578/) (Zhao et al., EMNLP 2026)

• Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory (Zhao et al., EMNLP 2026)

ACL

• Qinlin Zhao, Xiaoyuan Yi, Jing Yao, Muhua Huang, Defu Lian, and Xing Xie. 2026. Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 58592–58615, Budapest, Hungary. Association for Computational Linguistics.