Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models

Aldo Carranza, Rezsa Farahani, Natalia Ponomareva, Alexey Kurakin, Matthew Jagielski, Milad Nasr


Abstract
We address the challenge of ensuring differential privacy (DP) guarantees in training deep retrieval systems. Training these systems often involves the use of contrastive-style losses, which are typically non-per-example decomposable, making them difficult to directly DP-train with since common techniques require per-example gradients. To address this issue, we propose an approach that prioritizes ensuring query privacy prior to training a deep retrieval system. Our method employs DP language models (LMs) to generate private synthetic queries representative of the original data. These synthetic queries can be used in downstream retrieval system training without compromising privacy. Our approach demonstrates a significant enhancement in retrieval quality compared to direct DP-training, all while maintaining query-level privacy guarantees. This work highlights the potential of harnessing LMs to overcome limitations in standard DP-training methods.
Anthology ID:
2024.naacl-long.217
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3920–3930
Language:
URL:
https://aclanthology.org/2024.naacl-long.217
DOI:
Bibkey:
Cite (ACL):
Aldo Carranza, Rezsa Farahani, Natalia Ponomareva, Alexey Kurakin, Matthew Jagielski, and Milad Nasr. 2024. Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3920–3930, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models (Carranza et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.217.pdf
Copyright:
 2024.naacl-long.217.copyright.pdf