Xueming Tang

Author directory

2026

Membership inference attacks (MIAs) are increasingly used to audit whether protected examples were included in language model fine-tuning. Yet for fine-tuned LLMs, high attack AUC alone is not reliable evidence of data use: a score may separate members from non-members due to exposure, but also due to length, base model difficulty, typicality, or domain alignment. We propose DeMIA, a diagnostic membership inference framework that learns separate channels for membership scoring and difficulty modeling using likelihood features from the base and target models. DeMIA combines membership prediction with matched-pair ranking, base-difficulty anchoring, and adversarial length control, encouraging the final score to preserve membership-relevant evidence while reducing direct shortcut reliance. Across GPT-Neo-125M, Qwen2.5-1.5B, and Llama-3.1-8B on Wiki10k, AGNews10k, and XSum10k, DeMIA outperforms scalar likelihood, reference-calibrated, neighborhood, calibrated Min-K, and DF-style baselines while maintaining strong low-FPR recall. The signal remains strong on matched hard subsets, while blind controls using length, base loss, or other features from the base model remain near chance. These results show that fine-tuned LLM membership signals can persist under the tested nuisance controls and should be evaluated with reliability diagnostics rather than raw AUC alone.