Large language models (LLMs) are increasingly being adopted in clinical practice, e.g., in medical diagnostic systems. However, their reliability is still a concern, as LLMs are prone to hallucinations and could lead to misdiagnosis, inappropriate treatment decisions, and potential harm to patients. To mitigate these risks, LLM-based medical diagnostic systems should incorporate safety guardrails capable of detecting suspicious predictions and flagging them for human review or further clinical assessment. In this work, we employ uncertainty quantification (UQ) as a safety mechanism within an enterprise diagnostic system. We evaluate 16 UQ methods and propose two task-specific modifications tailored to the medical diagnostic workflow. First, we show that methods based on semantic similarity between generated samples substantially benefit from replacing a general-purpose NLI model with a domain-specific model. Second, we adapt a supervised UQ method that relies on LLM internal states for the detection of erroneous generation from a black-box enterprise LLM accessible only through a text-generation API. Our approach leverages the internal states of an open-source, lightweight proxy white-box model. We demonstrate that this approach outperforms all other UQ methods in the considered setting.