Large language models (LLMs) have increasingly been used for zero-shot authorship verification, where a model must determine whether two texts share an author. We show that evaluated zero-shot prompting methods exhibit a conservative bias, often over-predicting different authorship. This behavior is especially pronounced when the texts of the same author differ in topic or content. Through qualitative and quantitative analyses, we find evidence that LLMs confuse differences in content with differences in authorship. To test whether this failure can be reduced, we introduce lightweight prompt interventions that instruct the model to focus on style rather than content. Across four datasets and five open-weight instruction-tuned LLMs from three model families, the best intervention improves same-author accuracy by 13.1 points while also increasing macro-F1 by 3.1 points on average across all 20 experimental settings. Our findings show that zero-shot authorship verification is sensitive to the separation of content and style and that small framing changes can meaningfully change the direction of LLM errors. Code is available at https://github.com/galactixx/when-llms-say-no.