Imagine two users interact with the same LLM. One is told it is the cutting-edge flagship; the other, an older, weaker model. They walk away with markedly different ratings of its usefulness and intelligence, yet they used the same model. In a controlled study, 162 participants each used one of six LLMs from two families across three collaborative tasks, after first viewing a landing page that matched, overstated, or understated their model’s true capability. This pre-interaction framing persisted through use. Oversold users rated the model more favorably than Undersold users before interaction; afterward, both groups had updated toward reality, but the impression gap did not fully close. Framing also changed how users worked with the model (with Oversold users prompting more directively, Undersold users more collaboratively), but the quality of what they produced together depended only on the model’s true capability, not on what users were told. Participants’ change in model impressions after use, measured across two impression measures, was not predicted by task performance (𝛽 = -0.01 and 0.11, both n.s.), but by whether the model met users’ expectations (𝛽 = 0.47 and 0.50, both p < .001) and how confident they felt working with it (𝛽 = 0.47 and 0.36, both p < .001). After interaction, users are still rating the pitch, not the product: user-elicited LLM evaluations, including the preference data driving public leaderboards, measure expectation management at least as much as the model itself.

The growing capabilities of AI models are leading to their wider use, including in safety-critical domains. Explainable AI (XAI) aims to make these models safer to use by making their inference process more transparent. However, current explainability methods are seldom evaluated in the way they are intended to be used: by real-world end users. To address this, we conducted a large-scale user study with 85 healthcare practitioners in the context of human-AI collaborative chest X-ray analysis. We evaluated three types of explanations: visual explanations (saliency maps), natural language explanations, and a combination of both modalities. We specifically examined how different explanation types influence users depending on whether the AI advice and explanations are factually correct. We find that text-based explanations lead to significant over-reliance, which is alleviated by combining them with saliency maps. We also observe that the quality of explanations, that is, how much factually correct information they entail, and how much this aligns with AI correctness, significantly impacts the usefulness of the different explanation types.