Most existing harmful prompt detection methods rely on surface-level semantic representations, making them vulnerable to paraphrasing, prone to false positives, and limited in their ability to generalize to novel or adversarial prompts. Recent work has explored the use of model-internal signals, such as gradients or spectral features, to train classifiers that capture latent malicious behavior beyond surface semantics. However, these approaches primarily operate on local or isolated internal features and do not model the global structure of model behavior, which limits their robustness against subtle adversarial manipulations. In this work, we focus on robust harmful prompt detection under adversarial attacks. We propose SAFER, a simple yet effective method that models cross-layer attention signature flow by learning spectral correlation priors between attention heads and constructing a cross-layer attention-head spectral graph. This facilitates explicit modeling of structured attention propagation across layers, allowing the capture of both local and global attention dynamics and leading to improved robustness. SAFER achieves superior performance across multiple benchmark datasets for harmful prompt detection, while demonstrating significantly greater resilience to diverse adversarial attacks compared to prior approaches.