TextObfuscator: Making Pre-trained Language Model a Privacy Protector via Obfuscating Word Representations

.


Introduction
Pre-trained language models (PLMs) have achieved impressive performance on various NLP downstream tasks (Devlin et al., 2018;Brown et al., 2020; Qiu et al., 2020), but they also come with increased model size and significant computational requirements.In real-world applications, these large-scale models are often offered as an inference service (Altman, 2022).The service providers train PLMs for target tasks and deploy them on the cloud.Clients who lack high-computation resources can query these service with their input and obtains the desired responses (DALE, 2015).
Unfortunately, current inference services are plagued by serious privacy concerns (Lehmkuhl et al., 2021).Client data may contain sensitive information such as names, addresses, and even trade secrets, sharing such information with service providers compromises the privacy of clients.To address privacy risks, a naive solution is for clients to generate shallow representations on their devices and upload numerical representations to the cloud for subsequent inference, as shown in Figure 1.However, recent text reconstruction methods (Song and Raghunathan, 2020;Pan et al., 2020) have shown that word representations can be easily transformed into raw texts, indicating privacy risk remains.Inference service without privacy guarantees is not only unacceptable for clients but also illegal for service providers 1 .
Recent literature has proposed various methods to mitigate privacy leakage in representation.For example, Chen et al. (2022b) and Hao et al. (2022) have applied homomorphic encryption (Gentry, 2009) to transformer-based models, which enables computations to be performed on encrypted data.But homomorphic encryption often incurs significant computation time and communication costs (Gilad-Bachrach et al., 2016), making it impractical for real-world applications.Alternatively, several studies have adapted differential privacy (Lyu et al., 2020a;Hoory et al., 2021;Yue et al., 2021a) and adversarial training (Li et al., 2018;Coavoux et al., 2018;Plant et al., 2021) to reduce the privacy information contained in representations.However, in our scenario, the privacy information pertains to each word, and reducing word information in the shallow layer can harm subsequent inference, thus degrading performance (Jawahar et al., 2019), especially in token-level tasks.
In this paper, we propose TextObfuscator, a novel paradigm for privacy-preserving inference.
The key idea of our method is to learn private representations that obscure original word information while preserving original word functionality.Specifically, we find prototypes for each word and encourage functionally similar words close to the same prototype during training.Subsequently, random perturbations are applied to these clustered representations, which yields two key benefits.Firstly, it obscures original word information as the perturbed representations are indistinguishable from those clustered around them, making it harder for privacy attackers to reconstruct original words, thus protecting privacy.Secondly, it maintains the original word functionality as the perturbed representations stay within the same functional clusters, leading to improved performance.
To learn clustered representations, we have designed different methods to find suitable prototypes for token and sentence classification.For tokenlevel tasks, each word is assigned a label that serves as a prototype indicator (Snell et al., 2017).But for sentence-level tasks, there is no explicit prototype indicator for words.Therefore, the clustering algorithm is used for word assignment.Based on clustering results, we take the cluster centers as prototypes and assign semantically similar words to the same prototype.However, semantic-based clustering may lead to keywords from different classes being clustered together, hindering target tasks.For example, if "good" and "bad" play the same role, the sentiment of a sentence will be ambiguous.To address this, we utilize TF-IDF to identify keywords from different classes and use these keywords to redivide the clustering results.Our codes are publicly available at https://github.com/xzhou20/TextObfuscator.
Our contribution can be summarized as follows: • We propose TextObfuscator, a novel representation learning method for privacy-preserving inference by obfuscating representations.
• We propose to combine semantic and task information for finding prototypes, which leads to improved performance.
• We evaluate TextObfuscator on several NLP tasks including token and sentence classification, and demonstrate its effectiveness in protecting privacy while improving performance.

Inference as Service
Suppose a client wants to query the inference service without leaking privacy, client can perform acceptable computations on their device (such as intelligent chips, smartphones, and personal computers) to obtain the word representations H = f θc (X), where f θc is the client model released by service provider.Then numerical H instead of plain text X are uploaded to the cloud.The PLM f θs deployed on the server performs subsequent compute-intensive inference Y = f θs (H) and sends predictions Y back to the client.In this scenario, only representations are shared with service providers, avoiding the leakage of text.

Privacy Threats
Privacy threats in the inference phase mainly come from service providers, who have access to the client model f θc , server model f θs and client's word representation H.Recently studies (Song and Raghunathan, 2020) have shown that the word information in the representation is sufficient to reconstruct the original text.For example, the shallow representations are usually similar to their embedding, privacy attackers can compare the Server Model (Large) Step1: Find Task-Related Prototypes Client Model (Small)  !"#$ +  %&'#!()

Step2: Private Representation Training
Figure 2: Overview of the proposed method.In the first step, we find the task-related prototype using semantic and task information.Prototypes get initialized, and each word is assigned to a prototype.In the second step, we use word representation from the client model to calculate cluster loss, which encourages functionally similar representations clustered together.Then random perturbations are applied to representations to mislead privacy attackers, and we send these perturbed representations to the server model for task loss.Finally, we optimize the small client and large server models via cluster loss and task loss.
representation and embedding matrices to identify the most similar word.Furthermore, service providers can generate word representations via client model and train a powerful inversion model to directly transform the representations back into the original text X = f θ inv (H), even if privacypreserving methods have been applied on H.The challenge in private inference lies in ensuring f θ inv cannot learn useful word information from H to reconstruct X, while that the information in H is sufficient for subsequent inference.
3 Our Method

Overview
In this section, we present TextObfuscator, a novel framework for privacy-preserving inference.Our method learns private representation from a new perspective, which aims to obfuscate rather than reduce word information.The overall framework of TextObfuscator is shown in Figure 2. We first find prototypes for each word using semantic and task information, these prototypes are used to encourage functionally similar words to cluster together in the training phase.Then random perturbations are applied to each representation, making them indistinguishable from the surrounding clustered word representations.Even if privacy attackers get representation, they cannot establish the correct connection between the obfuscated representations and original words.Furthermore, these representations maintain original word functionality as they remain close to their prototype.Next, we introduce how to find prototypes and learn private representation.

Find Task-Related Prototypes
In step one, we introduce two crucial components.One is M(x i ) = p x i , which assigning word x i to its prototype p x i , refered as word assignment.The other is obtaining initial prototypes P = {p i } np i=1 , refered as prototype initialization.We enhance word assignment and prototype initialization from the semantic and task perspective, as the function of a word is not solely determined by its semantics, but also by its role in the target task.

Token-Level Task
In the token-level task, each word is assigned a label, which corresponds to the initial definition of the prototype (Snell et al., 2017;Ji et al., 2022).As the result, we take the label of the word as the indicator of prototype for token-level tasks.
Given a token-level dataset , we first use the client model f θc to traverse the dataset D t and obtain the representations {H i } N i=1 where H i = f θc (X i ).These contextual representations can provide semantic information for prototype initialization.Then we assign words with the identical label to the same prototype and take the average representation of words within a particular class as the initial prototype.Suppose there are k representations belong to the label c, the prototype p c of label c can be represented as: where h c j is the j-th representation of label c and k is the number of representations in label c.
In this way, we leverage the task information from the label to guide the word assignment M, and subsequently utilize the semantic information from the word representations to obtain the prototype initialization P = {p i } np i=1 , where n p denotes the number of labels.

Sentence-Level Task
Unlike token-level tasks, there are no natural prototype indicators for words in the sentence-level dataset.To tackle this problem, we perform a clustering algorithm on the representations, using the clustering results to indicate word assignment and prototype initialization.
To perform clustering, we need to prepare a representation for each word.Similar to the token-level task, we first use client model f θc to traverse sentence-level dataset D s and obtain {H i } N i=1 .For word x i that appears repeatedly in different contexts, we calculate the average of their representations to obtain the final word representation xi = 1 k k j=1 h x i , where h x i j is the j-th word representation of word x i and k is the number of words x i occurs in D s .Finally, we get X = {x i } nx i=1 where n x is the number of unique words in D s , and perform K-Means on X: the clustering algorithm assigns semantically similar words to the same cluster, thus completing the word assignment M. The centroid of the clusters is used as the prototypes initialization where n p here is the pre-defined number of clusters.However, it is not appropriate to assign all semantically similar words to the same cluster.For example, in sentiment analysis, the word representations of "good" and "bad" may be similar in representation space and are often assigned to the same cluster, but if they play the same role in a sentence, it can lead to ambiguity in the sentiment of the sentence.We refer to words that are highly relevant to specific classes as task-related words, and it is important to ensure that task-related words from different classes are assigned to different prototypes.
To identify task-related words for each class, we use the TF-IDF (Salton and Buckley, 1988), a numerical statistic that reflects a word's importance in a document.In this case, we treat all sentences within a class as one document and use TF-IDF to find the keywords for each class.Subsequently, the resulting keywords are used to re-divide M and update P, the algorithm of re-division is shown in Appendix A.3.

Private Representation Training
In the training phase, we use prototypes to encourage functionally similar word representations to be clustered in the representation space and apply random perturbation for preserving privacy.Clustering Loss.Given the input text X = {x i } n i=1 and word assigniment M, we first use client model f θc to get the word representation H = {h i } n i=1 , then use center loss (Wen et al., 2016) to make representation close to its prototype: where Furthermore, we also pull away the distance between different prototypes to prevent these prototypes collapse during training (Li et al., 2021a), thus enhancing task performance.The prototype distance loss is formulated as: where n p is the number of prototypes.We refer to L close and L away together as L cluster .Random Perturbation.We apply a random perturbation to each representation h i , which shifts the representation to another point near its prototype.Following Plant et al. (2021), we take the Laplace Noise as the random perturbation.The perturbed representations are sent to the server model f θs for subsequent computation: where Ŷ is the prediction results and ϵ is the hyperparameter to control the scale of noise.The perturbation is applied in both the training and inference phases.Because each perturbation is random, it is difficult for a privacy attacker to establish a link between the perturbed representation and the original word.Perturbed representations deviate from the original words but still serve original functions, thus preserving privacy while maintaining performance.
Overall Loss.The supervised task loss L task is joint learning in a multi-task learning manner, the overall loss for our model is: where  , 2020) also trains an inversion model like Inversion-Attack, but it runs in a multi-label classification manner and predicts a set of words in the sentence independent of their word ordering.

Defence Methods
We compare our TextObfuscator with three representative privacy-preserving methods and standard Fine-tune (Devlin et al., 2018).DPNR (Lyu et al., 2020b) uses differential privacy and word dropout to provide a privacy guarantee.CAPE (Plant et al., 2021) further adopts differential privacy and adversarial training to reduce privacy information in representation.SanText+ (Yue et al., 2021b) replaces the sensitive words in plain text based on differential privacy and word frequency.

Privacy Metrics
TopK is a token-level metric that measures the percentage of correct words in the attacker's top k predictions.RougeL (Lin, 2004) is a generation metric that measures the overlap between two sentences.We follow Gupta et al. (2022) and take it as a sentence-level metric to measure the coherence of attack results.Set is a metric specific to MLC-Attack, which quantifies the proportion of words in original sentence that are present in prediction set.Details of metrics are shown in Appendix A.2.

Experimental Settings
In our experiments, all methods are implemented based on roberta base (Liu et al., 2019).We divide the model into a smaller client model f θc with three transformer layers and a large server model f θs with the remaining nine transformer layers.
The privacy attack methods are all performed in the output representations of f θc , which will be shared with service providers and under the risk of privacy leakage.For the privacy defence methods, DPNR, CAPE, and our TextObfuscator are applied to the output representation of f θc , and SanText+ is applied to the input text directly.The implementation details and hyperparameters are shown in Appendix A.4.

Main Results
Table 1 shows the main results of our method and all baselines.We can observe that: (1) Representations without defence method are vulnerable to privacy attacks.In the absence of any privacy defence methods, all privacy attacks on Fine-tune are highly successful.Inversionattack even achieves 100% top-1 attack accuracy, indicating that privacy is fully compromised.
(2) Resisting Inversion-Attacks is key to protecting privacy.Most defence methods can resist KNN-Attack, it only achieves nearly 0 Top1 and Top5 attack accuracy for all tasks.In the case of MLC-Attack, the attack results are a set of disordered words that may contain redundancy.It is hard to reconstruct the correct sequence from such words.However, for Inversion-Attack, the representation is transformed into the original word one-to-one, and it achieves the highest attack accuracy, which is the most likely to compromise privacy.
(3) Previous defence methods achieve limited task performance.CAPE and DPNR show good privacy on sentence-level tasks, Inversion-attack on these methods only achieve about 5% top 1 attack accuracy on SST2 and AGNEWS.But for tokenlevel tasks which require richer word information, these methods not only degraded privacy but also suffered significantly from task performance.We speculate that these methods reduce the privacy information, i.e., word information, in the representation, which hinders the understanding of the sentence.There is an inherent contradiction between reducing word information on shallow representations and maintaining task performance, especially on token-level tasks.(4) With equal or better privacy, our proposed TextObfuscator shows a task performance improvement over baselines.The advantage of the TextObfuscator is that we do not reduce private information but rather obfuscate clustered representations, which misleads privacy attackers while still preserving functionality for each word representation, thus achieving better task performance while maintaining privacy.

Ablation Study
Effect of Different Components.To verify the effectiveness of the different components (L close , L away and random perturbation) in our method, we conduct a series of ablation experiments and show the results in Table 2.We can observe that: (1) Without cluster loss (L close and L away ), random perturbation alone is inadequate as a defence against privacy attacks.We speculate that the perturbation applied to unclustered representations can only provide limited obfuscation to the attacker.Most perturbed words still maintain a distance from other words, providing attackers the opportunity to distinguish between them.( 2 Effect of Clustering Algorithms.Sentence classification tasks require two additional processes, clustering and re-division algorithms.We conduct experiments on SST-2 to verify the performance and privacy impact of the cluster number and TF-IDF-based re-division.From experimental results shown in Figure 3, we can observe that redivision (KMeans-TFIDF) consistently improves task performance and privacy for all cluster numbers.Besides that, we find that a large cluster number can damage privacy (lower Top1 means better privacy), and a small cluster number can lead to a degradation in task performance.Therefore, a moderate cluster number is deemed to be optimal.

Visualisation
Representation Visualisation.To intuitively show the influence of the cluster loss and random From the visualization results in Figure 4, we can observe that, before perturbing (triangular point), functionally similar representations are clustered together while maintaining a certain distance from other clusters.After perturbing (round point), the representations are shifted and mixed with surrounding representations but remain within their respective functional clusters.Such representations obfuscate word information as they are indistinguishable from each other, but maintain word function as they still perform the same function in the representation space.We take NER as an example, perturbing the word representation of "John" may result in it being similar to another word, such as "Mike".However, a privacy attacker will only be able to establish a false association between the representation of "Mike" and the original word "John", thereby effectively protecting privacy.But for NER task, both words "John" and "Mike" serve the same role as "PER (Person)" and do not negatively impact the model's ability to classify them.These visualization results provide empirical evidence for the principles and effectiveness of TextObfuscator.
Attack Results Visualisation.To intuitively show the effectiveness of our privacy-preserving method, we visualize the results of privacy attacks for one sample from OntoNotes5.As shown in Table 3, we can observe that the attack results on TextObfuscator are largely unreadable, with only some high-frequency words "the" and wrong words  such as "Putin" being recovered.such as people, places, and time that may contain privacy have not been recovered correctly, indicating that our method is effective in protecting privacy.

Related Work
The high performance and computational cost of PLMs have accelerated the development of inference services (Soifer et al., 2019;Pais et al., 2022).These services enable clients to perform compute-intensive PLM inference in the cloud by uploading personal data, which brings convenience but also raises concerns about privacy (Zhang et al., 2021;Liu et al., 2020a).
In order to mitigate privacy leakage, many sought to upload representations that have been privatized by privacy-preserving technologies instead of the original text to the cloud (DALE, 2015).One method is to encrypt the representation, using either homomorphic encryption (Chen et al., 2022b) or a customized encryption protocol (Hao et al., 2022) to enable computations to be performed on the encrypted representation.Encryption-based methods often require high computation time and communication costs (Gilad-Bachrach et al., 2016) and may not be practical for real-world applications.Therefore, we did not compare this method in our experiments.Another method is to use Differential privacy (Xu et al., 2020;Lyu et al., 2020a;Yue et al., 2021a;Hoory et al., 2021) and adversarial training (Coavoux et al., 2018;Plant et al., 2021;Chen et al., 2022a) to learn private representation, which reduces privacy attributes in representation.Applying these works to reduce word information leads to limited performance, as the word information in the shallow layer is important for subsequent inference.Our method proposes to obfuscate word information while maintaining word functionality, thus providing better performance and privacy.Recently, Zhou et al. (2022a) propose TextFusion, which utilizes token fusion to hinder privacy attackers from training a targeted inversion model.We explore a stronger and more realistic setting than TextFusion, where the privacy attacker is the service provider itself.As the service provider is aware of TextFusion's defense strategies, they can design targeted privacy attack methods to disclose more sensitive information.We did not compare our method with TextFusion due to different settings.
In addition to NLP, there are also many works for protecting inference privacy in computer vision (Xiang et al., 2019;Osia et al., 2020;Liu et al., 2020b), However, most of these methods cannot be used directly in NLP because they only consider one single image, and we need to protect the privacy of a sequence of words.The popularity of transformer structures (Dosovitskiy et al., 2020) in computer vision may alleviate this situation, but the adaptation of these methods still requires further exploration.

Conclusion
In this paper, we propose TextObfuscator, a novel representation learning method for privacypreserving inference.The main idea of our method is to obfuscate word information and maintain word functionality.We achieve this by applying random perturbations to the clustered representations.The perturbed representations are indistinguishable from the surrounding representations but still around their functional clusters.To learn clustered representation, we find prototypes for each word and encourage the word representation to be close to its prototype.Additionally, we propose different methods to find prototypes for token-level and sentence-level tasks, utilizing semantic and task information.Through experiments on token and sentence classification tasks, we evaluate the effectiveness of TextObfuscator and provide further analysis of the principles of our proposed method.Overall, our results suggest that TextObfuscator is a promising method for preserving inference privacy.

Limitations
We summarize the limitations of our method as follows: (1) TextObfuscator was designed to protect word privacy in the inference phase, and we did not verify its ability to preserve other privacy attributes and training phase privacy.(2) Although we have done empirical experiments and visualizations to demonstrate the effectiveness of our method, a mathematical proof would enhance its privacy guarantees.(3) Our method requires more training steps than fine-tuning, resulting in an increased computational cost.

A.2 Privacy Metrics
In our experiments, we use three metrics to measure privacy.Next, we will describe these three metrics in a formulaic way.
TopK.Top-K accuracy is defined as the proportion of times that the real words is among the top K predictions made by the attack model, where K is a pre-defined parameter.Mathematically, it can be represented as: where N is the total number of representation, y i is the real word of representation i, p i is the predicted probability distribution of the attack model, and top k (p i ) is the set of top k words with highest probability for representation i.
RougeL (Lin, 2004).RougeL is a widely used metric to evaluate the quality of text summarization.So we do not describe the details of RougeL, but just state that we take the top1 word of the attack results to compose the sentences for calculating RougeL.
Set.The attack results of MLC-Attack are unordered sets of words, a one-to-one metric like TopK cannot be used for this attack, so we use Set to measure the attack success rate of MLC-Attack.
Given that set A is different words in a sentence, the set B is the prediction results of MLC-Attack, the Set metirc can be represented as: This metric measures how many words in the original sentence are in the set of predicted results of the MLC-Attack.

A.3 Re-division Algorithm
As mentioned in Section 3.2.2, after we find the category-related words T = {t c } nc c=1 for each category using TF-IDF, we re-divide the word Assignment M and prototype Initialization P. The algorithm is inspired by constrained K-means clustering (Wagstaff et al., 2001), but we only apply it once after clustering.The process of re-division is shown in Algorithm 1.

A.4 Implementation Details
In this subsection, we describe the implementation details and the replication process of both attack and defence methods.All methods are based on Roberta base with 125 million parameters.Dataset and models are all loaded from huggingface2 .Our experiments are conducted on NVIDIA GeForce RTX 3090.
Details for Defence Methods.We reference publicly available code, implement DPNR3 , CAPE4 and SanText+5 ourselves.We also conduct a grid search on the hyperparameters to reproduce the baselines on our setting.For each defence method, we train 50 epochs on SST2, CoNLL2003 and Ontonotes5, 30 epochs on AGNews to guarantee convergence, the AdamW optimizer and a linear learning rate scheduler are used during training.
The default learning rate is 5e-5, we do not adjust the learning rate unless we encounter the case of non-convergence.For DPNR, we search noise scale ϵ on [0.05, 0.1, 0.5, 1, 5] and the word dropout rate on [0, 0.1, 0.3].For CAPE, we search the adversarial training weights λ on [0.01, 0.05, 0.1, 0.5, 1, 5] and noise scale ϵ on [0.05, 0.1, 0.5, 1, 5].For Santext+, we follow the author's setting and use GloVe (Pennington et al., 2014) to guide the word replacement, the probability of non-sensitive words to be sanitized p defaults to 0.3 and the sensitive word percentage w defaults to 0.9.We search the privacy parameter ϵ on [1,2,3].For TextObfuscator, we use the K-Means to cluster representation for sentencelevel tasks, and the number of clusters defaults to 100.We search the close loss weights γ 1 from [0.1, 0.5, 1] and away loss weights γ 2 from [0.1, 0.3, 0.5].Although the noise scale can also be adjusted, we found that the most commonly used parameter (ϵ=1) is sufficient, so we kept the noise scale constant in all experiments.We select the best performance and privacy (but prefer privacy) results from the experimental results to report.The best hyperparameters we tuned are shown in Table 5.Details for Attack Methods.In our implementation of the KNN-Attack, we employed the use of the embedding matrix of the roberta base to calculate the Euclidean distance between the client representation.When attack the CAPE and DPNR methods, which employ max-min normalization on the representation, we also applied the same normalization technique on the embedding matrix before calculating distance.For Inversion-Attack and MLC-Attack, the training data is generated by the client model to be attacked on the training set of the target task.We use the Roberta base model as the backbone for the inversion model and search the learning rate from [1e-4, 1e-5, 1e-6] and train 10 epochs to guarantee convergence.We take the words with a probability higher than 0.5 as the prediction result of MLC.

Figure 1 :
Figure 1: Illustration of the inference services and privacy risks.Although clients upload word representations instead of plain text to the cloud server, privacy attackers can still transform representations into original texts.Clients are still at privacy risk.

Figure 3 :
Figure 3: Privacy and task performance under different cluster numbers.Accuracy is used for performance and Top1 under Inversion-Attack is used for privacy.
Input text: President Bush called his attention to the matter during the Italian leader's visit here last week.Fine-tune KNN: President Bush his attention to matter during Italian leader 's visit here last week.Inversion: President Bush called his attention to the matter during the Italian leader's visit here last week.MLC: { Bush | President | visit | Italian | week | last | ' | s | the | called | here | during | to | his | .| this } Text Obfuscator KNN: ...... anybody ls <= our Israeliibble >ancial clinicians, Wednesday Sag Jin relocation teleport.Inversion: the The Putin the the the the the the the the Israeli the the the the the next year, MLC: { to | the | .| in, }Table 3: Results of privacy attack.Text in red represents successfully recovered words.Text in bold means privacy information.Attacks on TextObfuscator only recover meaningless words, no useful information is leakaged.

Figure 4 :
Figure 4: Visualization of representations.Obfuscated representations are indistinguishable from surrounding word representations but still remain in their clusters.

Representation Space Mike Input Text Prototype lives in Second Street John stays at Third Avenue
The attacker can be anyone who has access to the client's representation.Inversion-Attack (Höhmann et al., 2021)requires the attacker to train an inversion model, which directly transforms the client representation to a word in a one-to-one manner.The attacker can be the service provider with access to the client and server model to generate training data for the inversion model.MLC-Attack (Song and Raghunathan (Qu et al., 2021)ghting factors.Inspired byLi et al. (2021a), we perform the clustering algorithm at the beginning of each epoch to make clustering results more accurate.During the training phase, the client model and server model are optimized together by service providers.During the inference phase, the client performs lightweight inference using the client model, then shares obfuscated representations with service providers and potential privacy attackers.Aside from perturbations, the inference phase of our method is the same as standard PLMs, thus, we do not introduce additional inference time.We use three recently proposed text reconstruction methods for privacy attacks.KNN-Attack(Qu et al., 2021)computes the distance between each representation and public word embedding matrix and takes the nearest word in the embedding matrix as the attack result.

Table 2 :
Without the L away , some prototypes tend to collapse to one point, resulting in a decline in task performance but a privacy boost.Ablation Study on our method.KNN and Inversion-attack use Top1 accuracy.

Table 4 :
results of SST-2 tasks are tested on the official development set.The statistics of datasets in our experiments are shown in Table 4. Statistics of the datasets.

Table 5 :
Hyperparameters of best restults for defence methods.-means the hyperparameter is not used in this method.λ adv is the adversarial training weights for CAPE.ϵ n is the noise scale used in CAPE, DPNR and TextObfuscator.ϵ w and ϵ p are used in Santext+, representing the word dropout rate and privacy parameter, respectively.γ 1 and γ 2 are the close loss weight and away loss weight for TextObfuscator.