How much of large language model (LLM) performance in code analysis reflects memorization of training data rather than genuine reasoning? We introduce BLOCKBENCH, a contamination-controlled evaluation framework that separates memorization effects from analytical capability in smart contract vulnerability detection. We evaluate seven frontier models on 316 vulnerable contracts augmented with 322 semantic-preserving transformations and 100 clean contracts. We observe a substantial performance gap across data exposure levels: detection accuracy drops from 86.5% on likely-contaminated samples to 27.6% on post-cutoff contracts, and further to 0.8% on the most recently disclosed subset. A temporal analysis shows a strong exposure effect, where detection rates increase from 5.9% to 84.7% as contracts become more publicly visible, indicating that benchmark performance is heavily inflated by data contamination.

We further identify a pattern matching paradox, where models can often locate security-relevant code regions but fail to explain why they are exploitable, revealing a form of explanation unfaithfulness in code understanding. In contrast, traditional static analyzers show complementary limitations: Slither compiles only 6.6% of post-cutoff contracts, while Mythril compiles all inputs but fails to detect vulnerabilities. BlockBench generalizes to code understanding tasks where semantic equivalence can be verified, enabling more reliable evaluation under potential contamination.