Evan Hubinger


2024

pdf bib
Steering Llama 2 via Contrastive Activation Addition
Nina Rimsky | Nick Gabrieli | Julian Schulz | Meg Tong | Evan Hubinger | Alexander Turner
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

We introduce Contrastive Activation Addition (CAA), a method for steering language models by modifying their activations during forward passes. CAA computes “steering vectors” by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses. During inference, these steering vectors are added at all token positions after the user’s prompt with either a positive or negative coefficient, allowing precise control over the degree of the targeted behavior. We evaluate CAA’s effectiveness on Llama 2 Chat using multiple-choice behavioral question datasets and open-ended generation tasks. We demonstrate that CAA significantly alters model behavior, is effective over and on top of traditional methods like finetuning and system prompt design, and minimally reduces capabilities. Moreover, we gain deeper insights into CAA’s mechanisms by employing various activation space interpretation methods. CAA accurately steers model outputs and sheds light on how high-level concepts are represented in Large Language Models (LLMs).

2023

pdf bib
Discovering Language Model Behaviors with Model-Written Evaluations
Ethan Perez | Sam Ringer | Kamile Lukosiute | Karina Nguyen | Edwin Chen | Scott Heiner | Craig Pettit | Catherine Olsson | Sandipan Kundu | Saurav Kadavath | Andy Jones | Anna Chen | Benjamin Mann | Brian Israel | Bryan Seethor | Cameron McKinnon | Christopher Olah | Da Yan | Daniela Amodei | Dario Amodei | Dawn Drain | Dustin Li | Eli Tran-Johnson | Guro Khundadze | Jackson Kernion | James Landis | Jamie Kerr | Jared Mueller | Jeeyoon Hyun | Joshua Landau | Kamal Ndousse | Landon Goldberg | Liane Lovitt | Martin Lucas | Michael Sellitto | Miranda Zhang | Neerav Kingsland | Nelson Elhage | Nicholas Joseph | Noemi Mercado | Nova DasSarma | Oliver Rausch | Robin Larson | Sam McCandlish | Scott Johnston | Shauna Kravec | Sheer El Showk | Tamera Lanham | Timothy Telleen-Lawton | Tom Brown | Tom Henighan | Tristan Hume | Yuntao Bai | Zac Hatfield-Dodds | Jack Clark | Samuel R. Bowman | Amanda Askell | Roger Grosse | Danny Hernandez | Deep Ganguli | Evan Hubinger | Nicholas Schiefer | Jared Kaplan
Findings of the Association for Computational Linguistics: ACL 2023

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user’s preferred answer (“sycophancy”) and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.