Measuring the value orientations of LLMs is helpful for diagnosing model risk and understanding their misalignment behavior. However, existing value benchmarks are often AI-generated and lack validity verification, leading to noisy and unreliable results. Inspired by Psychometrics, we propose a novel Subset Item Response Theory (S-IRT) model for assessing LLMs’ values under any value theories. S-IRT directly models the subset of values jointly expressed in each LLM response, rather than reducing responses to binary outcomes as in traditional IRT-based ability assessment. Meanwhile, it jointly estimates the LLM’s latent values and captures the relative priorities among them. Based on it, we further design a tailored information reward that estimates the information gain of each item in reducing uncertainty about the evaluated model’s value, and thus helps select a small set of high-quality items from existing benchmarks. Extensive experiments on 23 popular LLMs and three value systems (21 value dimensions) demonstrate S-IRT can be applied to multiple value systems simultaneously and improve evaluation reliability and validity. The information reward also helps reduce the number of questions to as few as 200 while preserving comparable evaluation effectiveness.

As large language models (LLMs) are gradually integrated into human daily life, assessing their underlying values becomes essential for understanding their risks and alignment with specific preferences. Despite growing efforts, current value evaluation methods face two key challenges. C1. Evaluation Validity: Static benchmarks fail to reflect intended values or yield informative results due to data contamination or a ceiling effect. C2. Result Interpretation: They typically reduce the pluralistic and often incommensurable values to one-dimensional scores, which hinders users from gaining meaningful insights and guidance. To address these challenges, we present Value Compass Benchmarks, the first dynamic, online and interactive platform specially devised for comprehensive value diagnosis of LLMs. It (1) grounds evaluations in multiple basic value systems from social science; (2) develops a generative evolving evaluation paradigm that automatically creates real-world test items co-evolving with ever-advancing LLMs; (3) offers multi-faceted result interpretation, including (i) fine-grained scores and case studies across 27 value dimensions for 33 leading LLMs, (ii) customized comparisons, and (iii) visualized analysis of LLMs’ alignment with cultural values. We hope Value Compass Benchmarks serves as a navigator for further enhancing LLMs’ safety and alignment, benefiting their responsible and adaptive development.