Modern language models are typically deployed using all transformer layers during inference, implicitly assuming that every layer contributes positively to downstream performance. Prior studies on layer skipping often focus on limited settings and commonly associate removable layers with specific transformer regions, particularly middle layers. However, whether single-layer skipping generalizes across diverse tasks, languages, and model architectures remains underexplored.

In this work, we conduct a multilingual and multi-task empirical study of single-layer skipping in modern instruct-tuned language models. We evaluate 9 text-only models spanning multiple model families and parameter scales, including Gemma 3, Llama 3.2, and Qwen 3, OLMo 2 without modifying model parameters. Our experiments systematically evaluate single-layer skipping across all layers on diverse benchmarks covering NLU, reasoning, bias evaluation, code generation, instruction following, and multilingual knowledge evaluation, including MMLU across 57 subject domains and 10 typologically diverse languages.

Our results show that the effect of single-layer skipping is highly context-dependent. We show that skipping a single transformer layer is an effective strategy for leveraging the capabilities of a language model: there is a high chance of achieving better benchmark performance while simultaneously reducing computation during inference. However, the optimal skipped layer varies depending on model architecture, task, domain, and language. Beneficial skipped layers occur most frequently in the second half of the network, though they also emerge in early layers in some cases. We further observe that performance improvements are more common in structured prediction tasks, whereas degradations more frequently occur in tasks requiring precise generation.