Ziyuan Chen
Author directory2026
Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models
Ziyuan Chen | Yujin Jeong | Tobias Braun | Anna Rohrbach
Findings of the Association for Computational Linguistics: EMNLP 2026
Ziyuan Chen | Yujin Jeong | Tobias Braun | Anna Rohrbach
Findings of the Association for Computational Linguistics: EMNLP 2026
As text-to-image (T2I) diffusion models become increasingly deployed in real-world applications, concerns about backdoor attacks have gained significant attention. Prior works largely focus on models with a single text encoder, leaving recent multi-encoder architectures, e.g., Stable Diffusion 3 (SD 3), underexplored. In this paper, we study SD 3, which uses three distinct text encoders, and systematically analyze its text-encoder-based backdoor vulnerabilities. To understand each text encoder’s role in backdoor attacks, we define four target categories spanning from global-level to entity-level semantics and identify the minimal encoder subset required for each attack type. Furthermore, we propose Multi-Encoder Lightweight aTtacks (MELT) and validate MELT’s effectiveness and parameter efficiency. MELT matches full encoder fine-tuning attack performance while using fewer than (0.2%) of its parameters, demonstrating that effective backdoors can be implanted under limited parameter budgets.