Yansu Wang

Author directory

2026

Backdoored generative LLMs can appear benign on normal inputs yet switch to unsafe or target-specific behavior when hidden triggers are activated, making request-level runtime detection a useful complement to model repair and offline auditing. Existing limited-black-box detectors typically rely on a single decoding trace, while text-only probing methods reveal susceptibility but often depend on specific probes or model-level audit goals. We introduce ProbeShift, a limited-black-box paired-probing framework that treats detection as repeated measurement of the same request under controlled prompt variations. It compares a baseline generation with templated probing generations and aggregates confidence dynamics, task-conditioned behavior cues, and paired response shifts. On safety-oriented BackdoorLLM DPA settings with LLaMA-2-7B-Chat, ProbeShift achieves remarkable performance: whilst maintaining high average recall rate of 97%, it reduced the FPR by 81% on average compared to the SOTA method ConfGuard based on single-track detection. These results show that paired probing yields a more stable and practical runtime signal than single-trace monitoring alone.