Industrial recommendation training increasingly relies on decentralized multi-GPU infrastructure, yet sparse embedding tables remain a major bottleneck due to significant storage, lookup, and communication overhead. Most existing works typically optimize only one of these bottlenecks or are designed for parameter server architectures, making them less effective in industrial decentralized training stacks. To tackle these challenges, we propose MixEmb, a practical framework for efficient decentralized embedding training through importance-aware quantization and caching. We first design a novel importance metric, i.e., embedding score, which can distinguish the contribution of embedding vectors based on spatial heterogeneity and temporal dynamics. Guided by embedding scores, MixEmb employs mixed precision quantization and hierarchical caching at fine-grained vector level. Besides, our key finding is that quantization and caching decisions are tightly coupled. Instead of naive combination, we delicately integrate them into a unified training framework through coupled co-design. A theoretical convergence analysis is also provided for MixEmb. We implement and evaluate MixEmb in real-world training scenarios. Across three workloads, MixEmb attains up to 1.55× speedup and 3.54× compression ratio compared to state-of-the-art methods.

Industrial-grade distributed training of vision-language models (VLMs) remains far less efficient than that of unimodal LLMs. Existing solutions either follow a monolithic design that assigns uniform parallelism to heterogeneous modules or adopt a disaggregated deployment that separates modules while executing them as a batch-synchronized pipeline. In this paper, we highlight that the above solutions are still not sufficient, and VLM training can be further decoupled. To this end, we present FlowTrain, a flow-based decoupled training framework that reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool. The encoder and backbone can progress independently over a global virtual address space. Since this execution decoupling fundamentally changes the optimization objective of allocation and scheduling, FlowTrain further introduces a heterogeneous parallel allocator that assigns module-specific parallelism strategies by solving a throughput matching problem. The dynamic packing scheduler is used to construct balanced microbatches at runtime according to the actual LLM-side computation cost. Extensive experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7× throughput improvement, narrowing the efficiency gap to LLM-only training.