This paper describes the development and validation of an automated scoring model for open-ended chatbot-based conversational speech among English learners in the Duolingo learning app. The model strongly predicted human ratings, produced reliable scores, and showed concurrent validity with Duolingo English Test speaking items, demonstrating stealth proficiency assessment at scale.