Chain-of-thought (CoT) reasoning has become a default paradigm across reasoning-intensive tasks and is widely assumed to be, at worst, harmless. Recent efficient-reasoning work treats unnecessary CoT purely as a cost to be reduced. We show that in boundary-dense classification tasks such as content moderation and toxicity detection, CoT is instead actively harmful. We identify the overthinking effect: extended reasoning amplifies weak surface signals into false evidence of violation, inflating false positive rates on ambiguous non-violating samples while its gains on ambiguous violating samples mask the harm in aggregate metrics. To address this, we propose Oreki, an adaptive CoT framework that uses K-fold cross-probing to label per-sample reasoning necessity and fine-tunes on a mixture of full-reasoning and reasoning-skipped formats. Across four base models on video moderation and multi-label toxicity detection, Oreki preserves CoT’s recall gains while suppressing its false positive inflation, outperforming both Always-CoT and Never-CoT. On toxicity detection, it cuts low-agreement false positive rate from 0.467 to 0.257 while raising overall F1 from 0.792 to 0.851.