2026

Static bias benchmarks score a model on isolated questions, yet LLMs that answer safely at first can still reveal bias as a conversation unfolds. We frame bias evaluation as multi-turn conversational elicitation and learn an interviewer that probes a target LLM, conditioned on a demographic group, a bias category, and a conversational strategy. The interviewer is trained with supervised fine-tuning on synthetic dialogues and refined with preference optimization on conversation-level feedback. Across all models we test, bias grows as the conversation continues, exposing failures that single-turn evaluation misses. Training the interviewer matters: it surfaces substantially more bias than the same model prompted with identical instructions, and preference optimization pushes it further. Multi-turn elicitation and single-turn search turn out to be complementary: at matched query budgets they surface similar amounts of bias, but conversation uncovers more severe cases. Human annotations corroborate these judgments. Our results argue that bias evaluation of conversational AI should move beyond single-turn probes.