DP-GTR: Differentially Private Prompt Protection via Group Text Rewriting

Mingchen Li, Heng Fan, Song Fu, Junhua Ding, Yunhe Feng


Abstract
Prompt privacy is crucial, especially when using online large language models (LLMs), due to the sensitive information often contained within prompts. While LLMs can enhance prompt privacy through text rewriting, existing methods primarily focus on document-level rewriting, neglecting the rich, multi-granular representations of text. This limitation restricts LLM utilization to specific tasks, overlooking their generalization and in-context learning capabilities, thus hindering practical application. To address this gap, we introduce DP-GTR, a novel three-stage framework that leverages local differential privacy (DP) and the composition theorem via group text rewriting. DP-GTR is the first framework to integrate both document-level and word-level information while exploiting in-context learning to simultaneously improve privacy and utility, effectively bridging local and global DP mechanisms at the individual data point level. Experiments on CommonSense QA and DocVQA demonstrate that DP-GTR outperforms existing approaches, achieving a superior privacy-utility trade-off. Furthermore, our framework is compatible with existing rewriting techniques, serving as a plug-in to enhance privacy protection. Our code is publicly available at anonymous.4open.science for reproducibility.
Anthology ID:
2025.findings-emnlp.83
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2025
Month:
November
Year:
2025
Address:
Suzhou, China
Editors:
Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1573–1585
Language:
URL:
https://aclanthology.org/2025.findings-emnlp.83/
DOI:
Bibkey:
Cite (ACL):
Mingchen Li, Heng Fan, Song Fu, Junhua Ding, and Yunhe Feng. 2025. DP-GTR: Differentially Private Prompt Protection via Group Text Rewriting. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1573–1585, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):
DP-GTR: Differentially Private Prompt Protection via Group Text Rewriting (Li et al., Findings 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.findings-emnlp.83.pdf
Checklist:
 2025.findings-emnlp.83.checklist.pdf