Strong large language models (LLMs) often achieve high scores on many benchmarks, yet can fail on closely related, sometimes easier, problems near their apparent capability boundary, a phenomenon known as jagged intelligence that reveals reasoning weaknesses masked by aggregate scores. Existing evaluations of jagged intelligence rely on manually designed benchmark-adjacent rewrites, which are hard to scale and quickly saturate as models improve. To address this challenge, we propose Automated Weakness Exploration (AWE), a fully automated framework for constructing and maintaining jaggedness evaluations. Using AWE, we construct JaggedBench with 1,142 verified capability-adjacent variants and evaluate 24 state-of-the-art models. On the primary evaluation set AIME 2025, benchmark-adjacent rewrites induce an average OG-to-RW relative drop of approximately 25% in Avg@k across 24 models, with consistent degradation observed across math, code, science, and knowledge domains. We identify 18 recurring failure modes. Our analysis shows that increasing reasoning intensity and scaling model size improve average performance and reduce failure frequency, yet the underlying failure-mode categories largely persist. Overall, these findings indicate that standard benchmark scores can present an overly optimistic view of reliability near the capability boundary. JaggedBench provides a more faithful and scalable lens for evaluating boundary robustness, enabling systematic diagnosis of boundary failures and longitudinal tracking of persistent weaknesses as models evolve. We release the benchmark, code, and subsequent updates at https://jaggedbench.github.io/.