Learning to simulate human users in interactive settings could advance the training of agents that serve humans, the evaluation of personalization systems, and research in the social sciences. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose a Turing Test approach for learning user simulator models, where a model is trained via reinforcement learning to produce responses indistinguishable from what the user could have said. Our discriminative Turing reward uses an LLM judge to score how indistinguishable a generated response is from the real user’s given the user’s history. Across two substantially different domains, namely conversational chat and Reddit forum discussion, we find that the Turing reward consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is an effective route to learning user simulators.

Large language models (LLMs) have seen increasing popularity in daily use, with their widespread adoption by many corporations as virtual assistants, chatbots, predictors, and many more. Their growing influence raises the need for safeguards and guardrails to ensure that the outputs from LLMs do not mislead or harm users. This is especially true for highly regulated domains such as healthcare, where misleading advice may influence users to unknowingly commit malpractice. Despite this vulnerability, the majority of guardrail benchmarking datasets do not focus enough on medical advice specifically. In this paper, we present the HeAL benchmark (HEalth Advice in LLMs), a health-advice benchmark dataset that has been manually curated and annotated to evaluate LLMs’ capability in recognizing health-advice - which we use to safeguard LLMs deployed in industrial settings. We use HeAL to assess several models and report a detailed analysis of the findings.