Autoregressive decoding remains a key bottleneck for LLM inference, due to its token-by-token nature. Speculative decoding is proposed to reduce latency by drafting multiple tokens and verifying them with fewer target-model passes. However, existing methods either incur auxiliary-drafter overhead or rely on brittle lexical span reuse. In structured tasks (e.g., code generation and text editing), we observe significant locality of reference and semantic redundancy for potential semantic context reuse. In light of this, we present Semantic Retrieval Speculative Decoding (SRSD), a training-free scheme that treats intermediate-layer hidden states as an in-context semantic index. At each decoding step, SRSD retrieves semantically similar past positions. Multiple candidates are merged into a compact tree and verified jointly in a single forward pass. SRSD requires no auxiliary models or external datastores. Across a variety of popular LLMs, SRSD achieves up to 5.42× throughput speedups over state-of-the-art training-free methods and parametric drafters. The code is available at https://github.com/Secbrain/SRSD.