This study evaluates the effectiveness of an AI-based system (Wilbur) for generating NAEP mathematics items. Results from this investigation indicate moderate success for content-focused items but substantial challenges for items intended to assess the NAEP Mathematical Practices. Transitioning from GPT-4 to GPT-5.2 improved item quality, though human revision remained essential.