2026

We present a framework for evaluating LLM-based clinical reasoning using conformance metrics from process mining. Four local LLMs were tested on EMR data from 16 vertigo patients across four prompting methods, with five repetitions per combination, yielding six uncertainty metrics. Reasoning traces were converted into event logs and analyzed with Heuristic Miner to derive external conformance to other patients’ reasoning patterns and internal conformance across repeated runs. Treating these as joint objectives, Pareto, Hypervolume, and TOPSIS analyses identified distinct evaluation scenarios, showing that conformance metrics can complement outcome-based uncertainty by revealing how LLMs reach their conclusions.