Arabic Dialect Identification (ADI) aims at automatically assigning an utterance—written or spoken—to a single variety, be it Modern Standard Arabic (MSA) or a variety of Dialectal Arabic (DA). To complement ADI, the Arabic Level of Dialectness (ALDi) has been proposed to model the different dialect levels that exist between MSA and Highly Dialectal text. However, current ALDi estimation models operate on text inputs only. This paper presents the first step toward extending ALDi to the speech modality. First, we use the automatically estimated ALDi score of a spoken utterance’s transcript as the silver-standard target score. We then compare the performance of two approaches at automatically predicting these silver-standard ALDi scores from spoken utterances: (1) a cascaded approach where automatic speech recognition is used to generate automatic transcripts that are then fed to a text-based ALDi estimation model, and (2) a direct ALDi estimation from speech. We use SADA—a dataset of 435 hours of manually transcribed Arabic speech—as the fine-tuning source data, and evaluate the two approaches on three datasets that cover different varieties of Arabic. Our results indicate that estimating silver-standard ALDi scores directly from speech is on-par or slightly worse than the cascaded approach for in-domain data. However, the direct approach generalizes better, especially for dialects where the fine-tuned ASR backbone of the cascaded approach has relatively large word-error-rates. This paper paves the way towards predicting ALDi scores directly from speech inputs.