Feiyan Chen
Author directory2026
One Trace Is Not Enough: Paired Probing for Runtime Backdoor Detection in Limited-Black-Box LLMs
Yansu Wang | Feiyan Chen | Rui Zhang | Kaizhu Huang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yansu Wang | Feiyan Chen | Rui Zhang | Kaizhu Huang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Backdoored generative LLMs can appear benign on normal inputs yet switch to unsafe or target-specific behavior when hidden triggers are activated, making request-level runtime detection a useful complement to model repair and offline auditing. Existing limited-black-box detectors typically rely on a single decoding trace, while text-only probing methods reveal susceptibility but often depend on specific probes or model-level audit goals. We introduce ProbeShift, a limited-black-box paired-probing framework that treats detection as repeated measurement of the same request under controlled prompt variations. It compares a baseline generation with templated probing generations and aggregates confidence dynamics, task-conditioned behavior cues, and paired response shifts. On safety-oriented BackdoorLLM DPA settings with LLaMA-2-7B-Chat, ProbeShift achieves remarkable performance: whilst maintaining high average recall rate of 97%, it reduced the FPR by 81% on average compared to the SOTA method ConfGuard based on single-track detection. These results show that paired probing yields a more stable and practical runtime signal than single-trace monitoring alone.