Recent work has shown that finetuned language models can predict human creativity ratings of metaphors, motivating claims that they can comprehend creativity in non-literal language. We question the interpretation of a previously reported performance of finetuned GPT-2 and RoBERTa and investigate the signals underlying their predictions. Using ridge regression probes, random baselines, token-order ablations, and representation-evolution features, we find that pretrained representations perform comparably to finetuned models. Randomly initialized models retain moderate performance when token and positional information is preserved, and token shuffling does not eliminate the predictive signal. These results suggest that performance is supported largely by surface-level cues, including sentence length, token identity, and token co-occurrence. We therefore reproduce the original results but challenge the interpretation, highlighting the importance of controlled reproducibility studies on LM-related findings and claims from other domains.