Safety evaluation of large language models (LLMs) increasingly relies on LLM-as-a-judge pipelines, but strong judges can still be costly to deploy at scale. We present a multi-agent judge that uses small language models (SLMs) in structured debates among critic, defender, and judge agents. To evaluate these systems, we introduce ForesightSafety Human-Annotated JailBench (FS-HAJailBench), a human annotated jailbreak benchmark with 11,100 labeled interactions spanning diverse attack methods and target models. FS-HAJailBench provides fine-grained ground truth for measuring both target-model robustness and judge reliability. On this benchmark, our SLM-based framework improves over matched small-model baselines and prior multi-agent judges, while approaching GPT-4o-level agreement with lower estimated unit cost. Ablation results show that three debate rounds provide the best accuracy-efficiency trade-off among the current Qwen3-14B variants. These results indicate that structured debate is a practical way to improve scalable LLM safety evaluation.