We introduce ZServe, an adaptive framework designed to optimize the deployment of large language models (LLMs) in production environments. ZServe tackles critical challenges in LLMs serving with three key innovations: (1) Resource-aware Parameter Mining, which automatically identifies optimal hardware configurations for various accelerators, eliminating the need for manual tuning while maximizing performance; (2) Task-aware Dynamic Reconfiguration, which seamlessly switches between SLO-centric and throughput-centric serving modes based on real-time workload characteristics without service disruption; and (3) A Unified Monitoring and Scaling Interface, which simplifies deployment management across heterogeneous infrastructures. In contrast to static frameworks like RayServe and Xinference, ZServe introduces dynamic runtime adaptation, leveraging novel queue handoff and memory-efficient state migration to switch serving modes without dropping in-flight requests. Experimental results with LLMs from 7B to 70B parameters show that ZServe achieves up to 45% higher throughput during peak periods while preserving latency targets during moderate traffic, demonstrating scalability across single-node and multi-node deployments.

We introduce a framework of Monte Carlo Tree Search with Double-q Dueling network (MCTS-DDU) for task-completion dialogue policy learning. Different from the previous deep model-based reinforcement learning methods, which uses background planning and may suffer from low-quality simulated experiences, MCTS-DDU performs decision-time planning based on dialogue state search trees built by Monte Carlo simulations and is robust to the simulation errors. Such idea arises naturally in human behaviors, e.g. predicting others’ responses and then deciding our own actions. In the simulated movie-ticket booking task, our method outperforms the background planning approaches significantly. We demonstrate the effectiveness of MCTS and the dueling network in detailed ablation studies, and also compare the performance upper bounds of these two planning methods.