Large language models (LLMs) remain vulnerable to jailbreak attacks that exhibit diverse prompts and rapidly evolving strategies. Existing defenses often operate at the level of prompts or predefined attack methods, limiting their ability to generalize to unseen attacks that share similar underlying structure but differ in wording. In this paper, we investigate whether seemingly different jailbreak attacks exhibit shared SAE feature co-activation patterns, and propose a motif-based framework that models attacks through these recurring patterns. Specifically, we first extract sparse autoencoder (SAE) features from jailbreak queries and discover motifs as recurring co-activation patterns via nonnegative matrix factorization. We show that motifs capture stable relationships across attack methods and generalize to unseen attacks. Building on this structure, we synthesize new jailbreak queries by composing motif-guided attack patterns. We then use the generated queries to construct preference-training data for Direct Preference Optimization (DPO). Experiment results show that SAE motifs serve as a stable and transferable interface for understanding and detecting jailbreak attacks, as well as enabling effective defenses that generalize beyond fixed prompt templates. Training target models with motif-guided synthetic data improves robustness to unseen attacks with modest changes in utility. Our work provides a practical methodology for advancing data-centric LLM safety through structured modeling of jailbreak attacks.

Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). First, we train a teacher model to quantify each sample’s degree of relying on shortcuts. Then, we encode this shortcut degree into a dummy class and use it to smooth the original ground truth labels, generating soft labels. These soft labels are used to train a more robust student model that reduces spurious correlations between shortcut features and certain classes. Extensive experiments on two NLU benchmark tasks via two language models demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy. Our code is available at https://github.com/ZiruiHE99/sle