With the success of reasoning models across complex natural language tasks, researchers in the information retrieval (IR) community have begun exploring how similar reasoning capabilities can be integrated into passage rerankers built on large language models (LLMs). These methods typically employ an LLM to produce an explicit, step-by-step reasoning process before arriving at a final relevance prediction. But does reasoning actually improve pointwise reranking? We investigate this question through controlled comparisons of Rank1, a pointwise reasoning reranker, with two non-reasoning counterparts. Across in-domain and out-of-domain datasets, we find that Rank1 generally underperforms Rank0, an otherwise equivalent non-reasoning pointwise reranker trained using the same backbone LLM and underlying training data as Rank1. To isolate the contribution of Rank1’s inference-time reasoning, we then replace its generated reasoning process with a fixed dummy sequence. Surprisingly, this variant, Rank1-NoReason, can also outperform Rank1. Analysis of Rank1’s score distributions provides a likely explanation: once Rank1 deems a passage relevant, it assigns nearly identical relevance scores, limiting its ability to distinguish degrees of relevance and ultimately harming its ranking accuracy. We release all code at https://github.com/nourj98/dont-overthink-reranking.