Penghui Li

Author directory

2026

Modern visual CAPTCHA has evolved from simple perception tests into real-world spatial reasoning challenges. It remains unclear whether state-of-the-art vision-language models can solve such tasks through their own reasoning ability. Existing benchmarks offer limited insight into this question as they often ignore intermediate reasoning steps, typically assess only final answers, and thus cannot know how models fail. To address this gap, we present CAPTCHA-X, the first unified evaluation framework for real-world CAPTCHA solving with reasoning steps, enabling automatic assessment of solving accuracy and reasoning quality. Using this framework, we show that reasoning substantially improves performance. Beyond evaluation, CAPTCHA-X exposes characteristic reasoning failure modes of vision-language models and reveals reasoning scaling laws that connect multiple dimensions of reasoning. Furthermore, evaluating pure vision generation models reveals an interesting phenomenon: generation models also fail at multi-step logical planning on images. These findings motivate our super-simple and reasoning-centric agent built solely on vision-language models, designed to identify and correct errors in intermediate reasoning steps before translating reasoning into executable actions.