Modern language models are typically deployed using all transformer layers during inference, implicitly assuming that every layer contributes positively to downstream performance. Prior studies on layer skipping often focus on limited settings and commonly associate removable layers with specific transformer regions, particularly middle layers. However, whether single-layer skipping generalizes across diverse tasks, languages, and model architectures remains underexplored.

In this work, we conduct a multilingual and multi-task empirical study of single-layer skipping in modern instruct-tuned language models. We evaluate 9 text-only models spanning multiple model families and parameter scales, including Gemma 3, Llama 3.2, and Qwen 3, OLMo 2 without modifying model parameters. Our experiments systematically evaluate single-layer skipping across all layers on diverse benchmarks covering NLU, reasoning, bias evaluation, code generation, instruction following, and multilingual knowledge evaluation, including MMLU across 57 subject domains and 10 typologically diverse languages.

Our results show that the effect of single-layer skipping is highly context-dependent. We show that skipping a single transformer layer is an effective strategy for leveraging the capabilities of a language model: there is a high chance of achieving better benchmark performance while simultaneously reducing computation during inference. However, the optimal skipped layer varies depending on model architecture, task, domain, and language. Beneficial skipped layers occur most frequently in the second half of the network, though they also emerge in early layers in some cases. We further observe that performance improvements are more common in structured prediction tasks, whereas degradations more frequently occur in tasks requiring precise generation.

Advancements in technology and the increased use of digital data threaten individual privacy, especially in speech containing Personally Identifiable Information (PII). Therefore, systems that can remove or process privacy-sensitive data in speech are needed, particularly for low-resource transcripts. These transcripts are minimally annotated or labeled automatically, which is less precise than human annotation. However, using them can simplify the development of de-identification systems in any language. In this study, we develop and evaluate an efficient speech de-identification system. We create an Indonesian speech dataset containing sensitive private information and design a system with three main components: speech recognition, information extraction, and masking. To enhance performance in low-resource settings, we incorporate transcription data in training, use data augmentation, and apply weakly supervised learning. Our results show that our techniques significantly improve privacy detection performance, with approximately 29% increase in F1 score, 20% in precision, and 30% in recall with minimally labeled data.

Indonesia is home to a diverse linguistic landscape, where individuals seamlessly transition between Indonesian, English, and local dialects in their everyday conversations—a phenomenon known as code-switching. Understanding and accommodating this linguistic fluidity is essential, particularly in the development of accurate speech recognition systems. However, tackling code-switching in Indonesian poses a challenge due to the scarcity of paired code-switching data. Thus, this study endeavors to address Indonesian-English code-switching in speech recognition, leveraging unlabeled data and employing a semi-supervised technique known as the machine speech chain. Our findings demonstrate that the machine speech chain method effectively enhances Automatic Speech Recognition (ASR) performance in recognizing code-switching between Indonesian and English, utilizing previously untapped resources of unlabeled data.

Despite being the world’s fourth-most populous country, the development of spoken language technologies in Indonesia still needs improvement. Most automatic speech recognition (ASR) systems that have been developed are still limited to transcribing the exact word-by-word, which, in many cases, consists of ambiguous sentences. In fact, speakers use prosodic characteristics of speech to convey different interpretations, which, unfortunately, these systems often ignore. In this study, we attempt to resolve structurally ambiguous utterances into unambiguous texts in Indonesian using prosodic information. To the best of our knowledge, this might be the first study to address this problem in the ASR context. Our contributions include (1) collecting the Indonesian speech corpus on structurally ambiguous sentences; (2) conducting a survey on how people disambiguate structurally ambiguous sentences presented in both text and speech forms; and (3) constructing an Indonesian ASR and meaning interpretation system by utilizing both cascade and direct approaches to map speech to text, along with two additional prosodic information signals (pause and pitch). The experimental results reveal that it is possible to disambiguate these utterances. In this study, the proposed cascade system, utilizing Mel-spectrograms concatenated with F0 and energy as input, achieved a disambiguation accuracy of 79.6%, while the proposed direct system with the same input yielded an even more impressive disambiguation accuracy of 82.2%.