Training Lightweight PII Text Detectors via Rubric-Guided Contrastive Data Synthesis

Shota Ekuni, Jie Zeng, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo


Abstract
The widespread use of large language models (LLMs) has increased the risk of leaking personally identifiable information (PII), making PII detection increasingly important. PII detection must process large text volumes efficiently, but labeled PII text is difficult to collect, share, and redistribute for privacy reasons. We aim to develop rubric-guided contrastive data synthesis for training lightweight PII text detectors from a few labeled seeds and to determine, through comparative experiments on PII benchmark datasets, whether the proposed method improves PII text detector classification performance. From about ten labeled examples, our method induces explicit criteria for distinguishing the two labels (rubrics), generates contrastive Sensitive/Non-sensitive sentence pairs, verifies and deduplicates them, and trains RoBERTa. Experiments on SPeDaC1 and SPY with Qwen3.5-9B and Gemma-4-26B-A4B showed that our method outperformed four baselines in all four settings and significantly improved Macro F1 over the Self-Instruct-style baseline in three of four settings. Overall, we developed rubric-guided contrastive data synthesis for training lightweight PII text detectors from a few labeled seeds and confirmed through comparative experiments on PII benchmark datasets that the proposed method improves classification performance. Our code is available at https://github.com/ekunish/emnlp2026-code.
Anthology ID:
2026.emnlp-industry.242
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Pepa Atanasova, Gholamreza Haffari, Minghan Wang
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4090–4104
Language:
URL:
https://aclanthology.org/2026.emnlp-industry.242/
DOI:
Bibkey:
Cite (ACL):
Shota Ekuni, Jie Zeng, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. 2026. Training Lightweight PII Text Detectors via Rubric-Guided Contrastive Data Synthesis. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 4090–4104, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Training Lightweight PII Text Detectors via Rubric-Guided Contrastive Data Synthesis (Ekuni et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-industry.242.pdf