Singular Value Decomposition (SVD) is a promising post-training compression technique for large language models (LLMs), as it is training-free, hardware-friendly, and preserves regular dense computation. However, existing SVD-based methods suffer from severe accuracy degradation under aggressive compression. This is mainly because hard rank truncation discards useful singular directions, while the two resulting SVD factors are usually compressed symmetrically despite their different roles, shapes, and quantization sensitivities. In this paper, we propose DELTA, an asymmetric low-rank compression method that decouples the latent heterogeneity between the two SVD factors. Under a fixed memory budget, the proposed method jointly determines the retained rank, factor-wise quantization bit-widths, and quantization directions, allowing more effective singular directions to be preserved with low-bit representations. To efficiently select compression configurations, we use an activation-aware reconstruction error as a layer-wise proxy objective. Experiments on multiple LLMs and datasets show that DELTA consistently outperforms existing post-training SVD-based compression methods, especially under high compression ratios. Compared with the strongest SVD-based baseline at the same compression ratio, DELTA reduces perplexity on WikiText-2 and C4 by up to 68.2% and 53.2%, respectively, and improves average zero-shot accuracy by up to 11 percentage points. These results demonstrate that asymmetric factor-aware compression can effectively mitigate the accuracy degradation of SVD-based LLM compression at high compression ratios. Code is available at https://github.com/BlacksheepJane/DELTA.

Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However, modern MoE inference remains heavily constrained by the rigid, whole-expert abstraction. Existing frameworks manage, schedule, or prune experts as atomic execution units, which fixes the optimization boundary too early and leaves fine-grained intra-expert computational redundancy underexplored. In this work, we present PCoMoE, a path-compositional execution framework that shifts MoE inference from coarse-grained expert selection to fine-grained path composition. PCoMoE incorporates a path-level formulation of expert computation, a compatibility-aware layer-wise pruning strategy to suppress low-value path combinations, and a hardware-friendly execution engine to exploit reusable sub-expert structures under strictly bounded overheads. Experimental results demonstrate that PCoMoE achieves up to a 1.31x end-to-end inference speedup while enhancing model accuracy by 10%. The code is available at https://github.com/gzyyy0/PCoMoE

Large language models (LLMs) have made significant progress in various NLP tasks, especially with the rise of long-context models that handle extended inputs. However, the growing size of the key-value (KV) cache in Transformer architectures imposes heavy memory demands during decoding, becoming a major bottleneck. Existing sparse attention methods aim to address this issue but face two main limitations: (1) they introduce high overhead when retrieving important tokens for attention, and (2) they often fail to significantly reduce memory usage in practice. This paper presents Seer, a simple yet effective system for fast and accurate LLM decoding on long-context sequences. Seer leverages semantic space representations to manage the KV cache more efficiently. It dynamically estimates cluster-level importance to reduce both computation and memory access, while preserving output quality. Through offline training, Seer learns semantic centroids to enable fast token importance estimation. To further improve efficiency, it integrates product quantization with sparse attention for KV cache compression and uses an asymmetric distance lookup table to reduce computational cost. Evaluation on a diverse set of LLMs and tasks shows that Seer reduces theoretical attention bandwidth by 40× with minimal accuracy loss. Seer achieves a 5.57× attention speedup and a 2× End-To-End speedup over full attention at 32K sequence length. Our code is available at https://github.com/Acaciasama/Seer.

Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of “Frequency-Guided Candidate Selection and Probability-Guarded Acceptance,” CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments.

The rapid advancement of large language models (LLMs) has exacerbated the memory bottleneck due to the widening gap between model parameter scaling and hardware capabilities. While post-training quantization techniques effectively reduce memory overhead, existing methods predominantly rely on static quantization strategies, which struggle to adapt to dynamic workloads. To address this, we propose FlexQuant, a dynamic precision-switching framework that optimizes the trade-off between inference speed and accuracy. Leveraging model perplexity entropy and Kullback-Leibler divergence, FlexQuant enables fine-grained, layer-wise mixed-precision quantization and dynamically adjusts bit-widths during each token generation. FlexQuant provides a comprehensive analysis of quantization strategies, introduces a precision requirement model for optimal switching, and implements efficient fine-grained precision management. Evaluations demonstrate that FlexQuant achieves a 1.3× end-to-end speedup across diverse language tasks with negligible accuracy loss introduced. This framework offers a flexible and adaptive solution for efficient LLM deployment.