Evaluating conversational agents typically requires human participants, a process that is costly and difficult to scale. LLM-based user simulators and LLM-as-a-judge evaluators offer a promising alternative, but how close they are to replacing real users remains unclear. We study this through a human user study (52 participants) and a simulated user study (four LLM-based simulators) applied to an experimental conversational sales agent, comparing LLM-as-a-judge assessments against human self-reported ratings. The results show that none of our simulators fully replicates human interaction patterns, and none of the LLM-based evaluators shows strong correlation with human ratings on real dialogues. We provide practical guidelines for combining automated and human evaluation, identifying key areas for improvement and highlighting when real-user studies remain essential.