Mingqian Zhang
Author directory2026
Branch Later, Compute Less, Think Better: Adaptive Depth Parallelism for Efficient Parallel Prefix-Tuning
Mingqian Zhang | Mouxiang Chen | Zhongxin Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Mingqian Zhang | Mouxiang Chen | Zhongxin Liu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Model-internal parallelization improves generation quality by introducing multiple parallel computation paths, but its additional latency and computational overhead hinder practical deployment. Recent studies reveal a functional division across model depth: shallow layers primarily encode input-conditioned features, whereas deeper layers are more involved in reasoning and generation. Since parallel paths process the same input and are optimized toward the same generation objective, their early representations are likely to overlap substantially, suggesting that early-layer parallelism may induce redundant computation. Motivated by this observation, we propose Adaptive Depth Parallelism (ADP), a framework that shares computation in shallow layers and activates parallel branches only after an adaptively selected bifurcation layer. We provide a sufficient-condition analysis for delayed branching and identify an empirical transition in inter-path cosine similarity across depth. This motivates using the stable minimum-similarity layer as a practical estimate of the bifurcation point. Experiments show that ADP achieves 104.83% of baseline performance while using only 39.4% of the computational load.