Machine unlearning aims to remove specified training data while preserving retained behavior, but gradient-based updates often induce a forgetting-retention tradeoff. This motivates our first-principles objective: optimize forgetting while keeping the retaining loss locally unchanged, formalized as retain invariance. We prove an equivalence under optimizer-induced geometry: the retain loss is locally invariant if and only if the update is orthogonal to the subspace spanned by retain gradients. Crucially, this geometry differs from Euclidean geometry because optimizer preconditioning rescales raw gradients before parameter updates. Based on this characterization, we propose Geometric-disentanglement Unlearning (GU), a plug-and-play projection module that filters any candidate forget update without modifying the underlying unlearning objective. GU projects the candidate update under the optimizer-induced metric to enforce first-order retain invariance and, under a trust-region constraint, gives the steepest forget direction among retain-preserving updates. Across 72 TOFU configurations, GU improves or preserves retain-set Extraction Strength in 91.7% and model utility in 86.1%, with gains transferring to MUSE and WMDP. Optimizer-metric GU outperforms its Euclidean counterpart on all four metrics, while running 5.1× faster than the per-sample-QR baseline. Code is available at https://github.com/Lemutisme/Geometric-Unlearning.

Large language models (LLMs) exhibit complementary strengths arising from differences in pretraining data, model architectures, and decoding behaviors. Inference-time ensembling provides a practical way to combine these capabilities without retraining. However, existing ensemble approaches suffer from fundamental limitations. Most rely on fixed fusion granularity, which lacks the flexibility required for mid-generation adaptation and fails to adapt to different generation characteristics across tasks. To address these challenges, we propose AdaFuse, an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation. Rather than committing to a fixed granularity, AdaFuse adjusts fusion behavior on the fly based on the decoding context, with words serving as basic building blocks for alignment. To be specific, we introduce an uncertainty-based criterion to decide whether to apply ensembling at each decoding step. Under confident decoding states, the model continues generation directly. In less certain states, AdaFuse invokes a diversity-aware scaling strategy to explore alternative candidate continuations and inform ensemble decisions. This design establishes a synergistic interaction between adaptive ensembling and test-time scaling, where ensemble decisions guide targeted exploration, and the resulting diversity in turn strengthens ensemble quality. Experiments on open-domain QA, arithmetic reasoning, and machine translation demonstrate that AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%.