Mateusz Kowalczyk

Author directory

2026

As large language models (LLMs) are increasingly deployed in real-world systems, red-teaming has become a critical tool for identifying safety vulnerabilities. However, existing approaches remain largely stateless and inefficient, repeatedly rediscovering similar failure modes for each new model. In this work, we propose a data-driven paradigm for red-teaming that enables the accumulation and reuse of adversarial knowledge across models. We introduce a framework for generating and collecting adversarial attacks enriched with structured metadata, including attack strategies, risk categories, and diagnostic feedback signals. Leveraging this resource, we demonstrate that metadata-aware retrieval significantly improves the efficiency of discovering successful attacks on unseen models, reducing reliance on purely generative search. To support further research, we release a large-scale dataset comprising >188k successful attacks across 8 models, together with rich metadata annotations.