Picture naming tests, which involve showing images and asking patients to name them, are widely used clinical and research tools to evaluate a patient’s ability to retrieve and produce words. One essential aspect of such tests is categorizing the type of semantic similarity between the image and the patient’s response, as classifying errors is important for treatment planning and prognosis. These tests are tedious to score and prone to inter-rater variability, both of which could be improved with automation via LLMs. This task highlights the clinical potential of LLMs, but also the need for robust statistical testing of LLM performance. In this work, we show that traditional evaluation methods can be misleading for complex experimental designs. Using generalized linear mixed effects models and conformal prediction for uncertainty quantification, we identify sources of variation in model performance and estimate the uncertainty of model predictions. We show that seemingly large differences in model performance from traditional evaluations are misleading, driven instead by item-level and category difficulty. Additionally, uncertainty in predictions varies substantially between semantic categories. These results suggest that for NLP evaluations with nested, clustered, or uneven category distributions, uncertainty quantification can be necessary to avoid misleading conclusions for high-stakes clinical tasks.