Saixing Gao
Author directory2026
MixEmb: Importance-Aware Quantization and Caching Co-Design for Industrial Decentralized Embedding Training
Zhida Jiang | Zhen Chen | Zhaolong Xing | Jiaxing Wang | Saixing Gao | Baopeng Yuan | Hao Wang | Yin Xu | Lin Liu | Dongyue Wang | Ke Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Zhida Jiang | Zhen Chen | Zhaolong Xing | Jiaxing Wang | Saixing Gao | Baopeng Yuan | Hao Wang | Yin Xu | Lin Liu | Dongyue Wang | Ke Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Industrial recommendation training increasingly relies on decentralized multi-GPU infrastructure, yet sparse embedding tables remain a major bottleneck due to significant storage, lookup, and communication overhead. Most existing works typically optimize only one of these bottlenecks or are designed for parameter server architectures, making them less effective in industrial decentralized training stacks. To tackle these challenges, we propose MixEmb, a practical framework for efficient decentralized embedding training through importance-aware quantization and caching. We first design a novel importance metric, i.e., embedding score, which can distinguish the contribution of embedding vectors based on spatial heterogeneity and temporal dynamics. Guided by embedding scores, MixEmb employs mixed precision quantization and hierarchical caching at fine-grained vector level. Besides, our key finding is that quantization and caching decisions are tightly coupled. Instead of naive combination, we delicately integrate them into a unified training framework through coupled co-design. A theoretical convergence analysis is also provided for MixEmb. We implement and evaluate MixEmb in real-world training scenarios. Across three workloads, MixEmb attains up to 1.55× speedup and 3.54× compression ratio compared to state-of-the-art methods.