KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks.

Tree-of-Thought (ToT) reasoning has significantly enhanced the problem-solving capabilities of Language Models by enabling the structured exploration of reasoning trajectories. However, the efficiency of these frameworks is often bottlenecked by search algorithms that struggle to navigate vast reasoning spaces at a reasonable inference cost. In this work, we address the challenge of optimizing LLM performance on mathematical reasoning tasks under limited token budgets. A key limitation of existing search strategies is that they apply the same exploration and selection rule throughout the entire search, which under a finite budget, often leads to token exhaustion with suboptimal or no candidate answers. Effective search instead requires different behavior at different stages. We introduce PACE (Phase-Aware Controller for Exploration), a strategy that adjusts its behavior based on the current state of the search: it prioritizes progress toward a first complete answer when none exists, and broadens exploration toward structurally distinct regions later on. Across GSM8K and Math500, two model scales, and token budgets from 3k to 10k, PACE consistently matches or improves over recent state-of-the-art baselines. On GSM8K with Llama-3.2-3B at a 3k budget, PACE delivers roughly a 56% relative accuracy improvement over the strongest tree-search baseline, and an early-stopping variant achieves comparable accuracy using about two-thirds of the tokens.

The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-training. We empirically observe that the activations of pre-trained LLMs exhibit low-rank property. Motivated by such observations, we propose CoLA and its memory-efficient implementation, CoLA-M, to replace these full-size layers with compute-efficient auto-encoders that naturally enforce low-rank activations throughout training. This fundamental architectural change eliminates the activation redundancy and significantly boosts model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by 2× and improves training throughput by 1.86× while maintaining full-rank level performance. CoLA-M further squeezes memory cost without sacrificing throughput, offering a pre-training approach with collectively superior parameter, computing, and memory efficiency. The LLMs produced are also 2× smaller, enabling faster inference with lower memory cost on resource-constrained platforms.