While in-context machine translation approaches lead to notable progress for low-resource languages, it is not yet clear what kinds of linguistic information in prompts can consistently and reliably lead to performance gains. We evaluate whether English semantic representations, specifically Abstract Meaning Representation (AMR) and its extension Uniform Meaning Representation (UMR), can improve LLM-based MT via prompting. Across 12 languages, 7 models from 5 families, and multiple prompting strategies, we find that conclusions about their effectiveness are highly sensitive to the setting. We further observe that prompts using UMR tend to outperform those using AMR, particularly in few-shot scenarios and for certain models. These results highlight the need for comprehensive evaluation for in-context learning work.