Chat assistants increasingly ask large language models to turn free-form user prose into structured, executable diagrams. These requests are noisy user-generated text: informal, hedged, jargon-mixed, incomplete, and often describing the target only indirectly—in workflow language rather than diagram vocabulary. Evaluation is hard because many distinct diagrams satisfy the same noisy request, making reference-based metrics too rigid where no ground truth exists, while reference-free LLM judges give weak, hard-to-reproduce diagnostics. We introduce QueryDiagramEval (QDE), a reference-free protocol scoring diagram generation directly against raw, noisy queries via two levels: deterministic compilation gates (syntax, rendering, diagram-type correctness), then a semantic-fidelity layer where a multimodal auditor is grounded in type-specific structural facts parsed from the code. On 255 annotated predictions, QDE aligns strongly with holistic human ratings (𝜌=0.715), outperforming an ungrounded LLM judge. We also release MermaidBench, a source-grounded benchmark of 1,150 query–gold pairs over 11 diagram types, four query-difficulty tiers, and a multi-turn track, with the gold held fixed to isolate robustness to input-side noise. Furthermore, we conduct experiments on eight frontier-model and harnesses to show that the our agentic framework reaches near-gold quality while failures concentrate in topology preservation, label grounding, and type-specific syntax.