Post-training quantization methods universally minimize element-wise reconstruction error, yet we show this is the wrong objective. A systematic correlation study on LLaMA models reveals that the representation kernel (the matrix of pairwise cosine similarities among calibration inputs) is a substantially stronger proxy for downstream task performance than weight MSE or output MSE. Building on this finding, we propose KernelQuant, a training-free PTQ framework comprising: KPQ, which selects weight-quantization scales via kernel-MSE grid search followed by least-squares error compensation; model-level Hadamard rotation; and KPC, a kernel-preserving per-channel activation clipping scheme. KernelQuant introduces zero inference overhead beyond QuaRot and calibrates in minutes rather than hours. On LLaMA 3B–70B, KernelQuant achieves state-of-the-art W4A16 perplexity and competitive W4A4 results. On reasoning benchmarks (MMLU), where 4-bit quantization causes a “reasoning cliff,” KernelQuant retains the highest fraction of FP16 accuracy among all evaluated baselines, both rotation-based (QuaRot, SpinQuant, FlatQuant) and element-wise, reversing the perplexity-based ranking and supporting the view that reasoning depends on the geometric relations the kernel preserves.

On-device LLM deployment routinely serves GGUF-quantized models whose safety is ver- ified at full precision rather than on the llama.cpp/GGUF stack end users actually run. We stress-test nine production mod- els across four GGUF levels against seven CHATBUG chat-format attacks on 520 Ad- vBench instructions, served through the same llama.cpp binary that powers Ollama, and report three deployment-relevant findings. (1) Q2_K vulnerability concentrates in the smallest on-device models: Llama-3.2-1B reaches 75.3% ASR-R at Q2_K (+26.9 pp vs. Q8_0) while Gemma-3-4B is essentially un- affected (+3.0 pp), so model choice dominates quantization-level choice within the on-device regime. (2) The vulnerability is a property of the deployed system: a subset reproduces end- to-end through Ollama (median |∆|= 2.7 pp). (3) Keyword- and moderator-based metrics diverge in magnitude: Llama-2 Mismatch-∅ shifts ASR-R +55.8 pp but ASR-M−1.7 pp at Q2_K, while Llama-2 Overflow-FS is con- firmed unsafe by both (+16.8 pp ASR-M). We release a one-page deployment-safety checklist and GGUF-native diagnostic probes, and iden- tify single-layer probes as insufficient cross- model predictors of ASR change, motivating per-layer llama.cpp instrumentation.

On-device fine-tuning enables privacy-preserving personalization of large language models, but mobile devices impose severe memory constraints, typically 6–12GB shared across all workloads. Existing approaches force a trade-off between exact gradients with high memory (MeBP) and low memory with noisy estimates (MeZO). We propose Memory-efficient Structured Backpropagation (MeSP), which bridges this gap by manually deriving backward passes that exploit LoRA’s low-rank structure. Our key insight is that the intermediate projection h = xA can be recomputed during backward at minimal cost since rank r ≪ d_in, eliminating the need to store it. MeSP achieves 49% average memory reduction compared to MeBP on Qwen2.5 models (0.5B–3B) while computing mathematically identical gradients. Our analysis also reveals that MeZO’s gradient estimates show near-zero correlation with true gradients (cosine similarity ≈0.001), explaining its slow convergence. MeSP reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on memory-constrained devices.

While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA’s B matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the B matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA’s full potential for effective LLM fine-tuning.