Zero-shot detectors for LLM-generated text aggregate token-level statistics uniformly, implicitly assuming that every position contributes equally to detection. We challenge this assumption and show that detection signal is highly uneven across token positions: positions of high contextual dependency, where the surrogate model relies on concentrated context, carry far more discriminative power than others. Using attention entropy to quantify this dependency, we empirically validate that aggregating only low-entropy tokens yields clearly separated human and LLM score distributions, whereas high-entropy tokens do not. Based on this finding, we propose Token Contextual Dependency (TCD), a plug-and-play framework that reweights token contributions in existing zero-shot detectors via attention entropy-based weights computed from dynamically selected layers. Experiments on state-of-the-art detectors across multiple benchmarks demonstrate consistent and meaningful improvements in detection performance. Our code is available at: https://github.com/RecursionM/TCD.