As large language models (LLMs) are deployed globally, creating pluralistic systems that can accommodate the diverse preferences and values of users becomes essential. We introduce EVALUESTEER, a human-validated benchmark to measure the steerability of LLM and reward models (RMs) to users’ value and stylistic preference profiles grounded in psychology and literature on human-LLM interaction. To address the gap in existing datasets that do not support controlled evaluations of RM steering, we synthetically generated 165,888 preference pairs – systematically varying pairs along 4 value dimensions (traditional, secular-rational, survival, and self-expression) and 4 style dimensions (verbosity, readability, confidence, warmth). We use EVALUESTEER to evaluate whether, given a user profile and a pair of candidate value and style laden responses, LLMs and RMs are able to select the output that aligns with the user’s preferences. Our results show that, when given the user’s full profile of values and stylistic preferences, the best models achieve <75% accuracy in choosing the correct response, in contrast to > 99% accuracy when only relevant style and value preferences are provided. RMs also exhibit systematic biases, favoring secular values and verbose, confident language, and prioritizing style alignment over value alignment when the two conflict. EVALUESTEER thus highlights the limitations of current RMs at identifying and adapting to relevant user profile information, and provides a challenging testbed for developing RMs that can be steered towards diverse human values and preferences.

There is a significant body of work looking at the ethical considerations of large language models (LLMs): critiquing tools to measure performance and harms; proposing toolkits to aid in ideation; discussing the risks to workers; considering legislation around privacy and security etc. As yet there is no work that integrates these resources into a single practical guide that focuses on LLMs; we attempt this ambitious goal. We introduce LLM Ethics Whitepaper, which we provide as an open and living resource for NLP practitioners, and those tasked with evaluating the ethical implications of others’ work. Our goal is to translate ethics literature into concrete recommendations for computer scientists. LLM Ethics Whitepaper distils a thorough literature review into clear Do’s and Don’ts, which we present also in this paper. We likewise identify useful toolkits to support ethical work. We refer the interested reader to the full LLM Ethics Whitepaper, which provides a succinct discussion of ethical considerations at each stage in a project lifecycle, as well as citations for the hundreds of papers from which we drew our recommendations. The present paper can be thought of as a pocket guide to conducting ethical research with LLMs.