Large language models frequently memorize specific factual associations from their training data, necessitating post-hoc removal of targeted knowledge. However, standard gradient-based unlearning techniques treat all parameter directions uniformly, often leading to catastrophic forgetting of general capabilities. We introduce **ERASE** (**E**igenspace-**R**egulated **A**daptive **S**elective **E**rasure), a geometry-aware unlearning framework that combines three complementary mechanisms: (1) a *spectral filter* that attenuates gradient updates along principal directions of retained model knowledge, confining forgetting pressure to low-interference subspaces; (2) *selective error-signal construction*, which restricts gradients to semantically meaningful token positions, avoiding diffuse collateral damage; and (3) an *adaptive Cross-Entropy Target (CET) loss with dynamic per-sample stopping*, which drives each forget example toward a target cross-entropy and halts on convergence, preventing both under- and over-forgetting. Our method requires no reference model, no retain-set loss, and no auxiliary task supervision during training. On TOFU within the OpenUnlearning evaluation framework, ERASE achieves the highest aggregate score across four model families and two forget splits, attaining the best privacy and near-original utility among all compared baselines; on the document-level MUSE-News benchmark, with forget spans recovered automatically via named entity recognition, it simultaneously attains the strongest forgetting and retention with the privacy score closest to the retrain oracle. Our code is publicly available at https://github.com/ItzikWaizman/ERASE.