As large language models (LLMs) are deployed globally, creating pluralistic systems that can accommodate the diverse preferences and values of users becomes essential. We introduce EVALUESTEER, a human-validated benchmark to measure the steerability of LLM and reward models (RMs) to users’ value and stylistic preference profiles grounded in psychology and literature on human-LLM interaction. To address the gap in existing datasets that do not support controlled evaluations of RM steering, we synthetically generated 165,888 preference pairs – systematically varying pairs along 4 value dimensions (traditional, secular-rational, survival, and self-expression) and 4 style dimensions (verbosity, readability, confidence, warmth). We use EVALUESTEER to evaluate whether, given a user profile and a pair of candidate value and style laden responses, LLMs and RMs are able to select the output that aligns with the user’s preferences. Our results show that, when given the user’s full profile of values and stylistic preferences, the best models achieve <75% accuracy in choosing the correct response, in contrast to > 99% accuracy when only relevant style and value preferences are provided. RMs also exhibit systematic biases, favoring secular values and verbose, confident language, and prioritizing style alignment over value alignment when the two conflict. EVALUESTEER thus highlights the limitations of current RMs at identifying and adapting to relevant user profile information, and provides a challenging testbed for developing RMs that can be steered towards diverse human values and preferences.

In this work, we tackle the challenge of embedding realistic human personality traits into LLMs. Previous approaches have primarily focused on prompt-based methods that describe the behavior associated with the desired personality traits, suffering from realism and validity issues. To address these limitations, we introduce BIG5-CHAT, a large-scale dataset containing 100,000 dialogues designed to ground models in how humans express their personality in text. Leveraging this dataset, we explore Supervised Fine-Tuning and Direct Preference Optimization as training-based methods to align LLMs more naturally with human personality patterns. Our methods outperform prompting on personality assessments such as BFI and IPIP-NEO, with trait correlations more closely matching human data. Furthermore, our experiments reveal that models trained to exhibit higher conscientiousness, higher agreeableness, lower extraversion, and lower neuroticism display better performance on reasoning tasks, aligning with psychological findings on how these traits impact human cognitive performance. To our knowledge, this work is the first comprehensive study to demonstrate how training-based methods can shape LLM personalities through learning from real human behaviors.