Text only partially determines prosodic realization, leaving residual variation that may align with speaker-level metadata such as regional accent or first-language (L1) background. However, because these labels are assigned to speakers, their predictive gains may overlap with broader speaker-associated structure rather than uniquely reflect category-level prosodic effects. We frame this as an attribution problem and propose a speaker-controlled audit using matched text-residual prosody predictors that differ only in access to metadata labels and speaker identity. The audit measures label utility beyond text, speaker utility beyond text, and the additional empirical utility of the label when speaker conditioning is already available. Across VCTK regional-accent labels and L2-ARCTIC L1 labels, metadata yields modest gains over text, while speaker conditioning yields larger gains and leaves little additional label utility under speaker control. Random speaker-group and label-perturbation controls further characterize the specificity and use of the supplied metadata. Importantly, speaker-held-out evaluation shows that metadata transfer to unseen speakers is limited and target-dependent, indicating that observed-speaker metadata gains do not by themselves establish speaker-independent category-level effects. These results position speaker-controlled auditing as a prerequisite attribution check before assigning category-level interpretations to metadata-conditioned predictive gains.