Focused Prefix Tuning for Controllable Text Generation

In a controllable text generation dataset, there exist unannotated attributes that could provide irrelevant learning signals to models that use it for training and thus degrade their performance. We propose focused prefix tuning (FPT) to mitigate the problem and to enable the control to focus on the desired attribute. Experimental results show that FPT can achieve better control accuracy and text fluency than baseline models in single-attribute control tasks. In multi-attribute control tasks, FPT achieves comparable control accuracy with the state-of-the-art approach while keeping the flexibility to control new attributes without retraining existing models.


Introduction
Controllable text generation aims to generate text associated with a specific attribute. For example, given an attribute TOPIC = sports and a prompt "There is," a model is supposed to generate a continuation whose TOPIC is sports, such as "There is a tennis match ...".
In datasets for the controllable text generation task, there exists the annotated attribute, and we call it an explicit attribute (e.g. the TOPIC attribute in the AGNews dataset). In addition to the explicit attributes, the datasets tend to have their own tendency. For example, up to 98% of training data pieces in the IMDb dataset exhibit "TOPIC = sci/tech", while up to 94% of training data pieces exhibit "SENTIMENT = negative". 1 We call the tendency an implicit attribute (e.g. the TOPIC attribute in the IMDb dataset).
The existence of the implicit attributes could degrade the performance in controlling for an explicit attribute when models are trained on the datasets. Since implicit attributes are of dataset-level and related to undesired explicit attributes, the probability 1 The models used for classification are from (Gu et al., 2022).

Model
Desired Attribute Implicit Attribute Relevance Relevance  Table 1: Relevance of texts generated by different models (e.g. DExperts and Vanilla Prefix Tuning) trained on IMDb dataset. We found a lower desired explicit attribute (e.g. SENTIMENT) relevance is related to a higher implicit attribute (e.g. TOPIC = sci/tech) relevance. The relevance is calculated by the classifier models in Sec. 4.2. of generating content with the implicit attributes is first likely to increase. When the text with the implicit attributes was generated, the probability of generating content with other undesired explicit attributes would increase, and the text with them might be generated next. As a result, as shown in Table 1, the model generates content with a high implicit attribute relevance but a low desired explicit attribute relevance (e.g. Vanilla Prefix Tuning (Li and Liang, 2021)). In contrast, if the model generates content with a low implicit attribute relevance, it will have a high desired explicit attribute relevance (e.g. DExperts (Liu et al., 2021). We call this phenomenon attribute transfer.
To mitigate the effect of the attribute transfer, we propose focused prefix tuning (FPT), which makes the generation focused on the desired explicit attribute. FPT uses specific and general prefixes to encode the explicit and implicit attributes, respectively. FPT combines the control power of the two prefixes via logits manipulation at inference time. Experimental results show that FPT achieved better control accuracy and fluency in single-attribute control tasks. In multi-attribute control tasks, FPT can achieve comparable performance with the state-ofthe-art approach. Moreover, we show, since FPT enables the training of each attribute prefix individually, we can incrementally add new attributes without retraining all prefixes.

Controllable Generation
Methods for controlling text generation have rapidly developed (Ficler and Goldberg, 2017;Rajani et al., 2019;Dathathri et al., 2020a;Madotto et al., 2020;Chan et al., 2021). Keskar et al. (2019) trained a large transformer model to generate contents conditioned on up to 55 attributes. However, the cost of training such a model is too high.

Prefix Tuning
Parameter-efficient fine-tuning (PEFT) methods, such as prompt tuning (Lester et al., 2021) have become particularly significant in driving various natural language processing tasks to reduce the high training cost. Prefix tuning (Li and Liang, 2021) is one of the PEFT methods that steers pretrained models (Radford et al., 2019;Lewis et al., 2020) by applying an additional continuous vector embedding before every activation layer. Qian et al. (2022) proposed a contrastive prefix tuning method that improves its performance by utilizing the relations between attributes. However, they focused only on attributes explicitly annotated and ignored the effect of implicit attributes.

Inference-time Methods
Inference-time methods, a lightweight approach without updating the parameters, have been used for controllable text generation (Yang and Klein, 2021;Dathathri et al., 2020a;Welleck et al., 2020;Holtzman et al., 2018). To enhance controllability, Krause et al. (2021) proposed a method to combine the computed classification probability distributions. Liu et al. (2021) found that directly applying probability distributions from language models is a simple but effective approach to control generated texts. Inspired by their work, we propose a method that uses probability distributions from language models to remove the effect of implicit attributes.

Focused Prefix Tuning
The task of controllable generation is, given a sequence of prompt tokens x <t and an attribute ATTR = val (e.g. TOPIC = sports), to generate a sequence of tokens as a continuation x that conforms to both the prompt and specified attribute.

Vanilla Prefix Tuning
In controllable text generation, a prefix can steer a pre-trained model parameterized by θ to generate texts under a specific attribute value ATTR = val. In particular, vanilla prefix tuning (Li and Liang, 2021) prepends a set of continuous vectors before every activation layer of the pre-trained transformer. The continuous vectors are referred to as the prefix H attr=val ϕ , which is parameterized by ϕ. During training, we freeze the pre-trained model's parameters θ and update only the prefix parameters ϕ to optimize the following objective: where D attr=val is the subset of the entire dataset D whose attribute ATTR is val. Following Li and Liang (2021), we initialize the prefix H ϕ with the activation of actual tokens from the pre-trained model's vocabulary.

Specific and General Prefixes
The prefix in vanilla prefix tuning captures an explicit attribute in a dataset by training it on the subset dataset D attr=val . To capture only implicit attributes while ignoring any explicit attributes, we propose to train another prefix on the entire dataset D. To distinguish the two prefixes, we refer to the prefix trained on D attr=val as a specific prefix and that trained on D as a general prefix.
The specific prefix is the same as the prefix in vanilla prefix tuning, so we still use Equation 1 to update its parameters. To update the general prefix's parameters, we optimize the following objective: where H genl ϕ ′ represents the general prefix, which is parameterized by ϕ ′ .

Inference-time Logits Manipulation
As shown in Figure 1, FPT suppresses the probability of words with implicit attributes in the generated text by combining logits z attr=val steered by the specific prefix and logits z genl steered by the general prefix via logits manipulation at inference time. For example, when generating text with the attribute TOPIC = sports, the probability of words with implicit attributes (e.g. "impossible" with SEN-TIMENT = negative) would be suppressed. During inference, at each step t, we first make two forward runs respectively with the specific and general prefixes to obtain their logits, z attr=val t and z genl t . Since z attr=val t encodes both the explicit and implicit attributes while z genl t encodes mostly the implicit attributes, we use a subtraction operation at the logits level to suppress the probability of words with implicit attributes: where α is a hyperparameter that can be interpreted as the strength for the control of implicit attributes. Following Liu et al. (2021), we respectively set α and α − 1 as the weight of z attr=val and z genl t to make the ratio of logits after the logits manipulation equal to 1.
To ensure the fluency of generated texts, we follow Liu et al. (2021) to use top-p filtering to remove the tokens that have low scores in advance before logits manipulation. In particular, we modify the logits produced by the specific prefix by calculating the top-p vocabulary V and setting all the logits outside V to −∞: Therefore, the logits manipulation in Equation 3 is updated as follows: The token at step t is then selected by ancestral sampling from P ′ (x t |x <t , ATTR = val).

Multi-attribute FPT
FPT is also applicable to the multi-attribute control task, where we aim to control multiple different attributes at the same time. Similarly, we first train the specific prefix for each attribute. Then, we adapt logits manipulation to the multi-attribute task as follows: where K is the number of different attributes. Each z attr i t is the combination of the logits from the corresponding specific prefix and general prefix. Since applying top-p filtering to every attribute could possibly result in an empty V , we apply the filtering only to the first attribute:  We also set up one variant of FPT: Contrastive FPT: Applying contrastive prefix tuning to train specific prefixes. We also set an ablated model that uses the logits of the frozen GPT-2 instead of the logits from the model guided by our general prefix.

Experimental Settings
Following previous work (Krause et al., 2021;Qian et al., 2022), we evaluated the models on a topic control dataset AGNews (Zhang et al., 2015) and a sentiment control dataset IMDb (Maas et al., 2011). We score the sentiment relevance using HuggingFace's sentiment analysis classifier (Liu et al., 2019) trained on 15 datasets. For scoring topic relevance, we trained the classifier that obtained comparable results to what was reported. Perplexity was used to evaluate text fluency. Bias (|relevance score − 50|) is how much the relevance of implicit attributes deviated from unbiased relevance (50). We set TOPIC = science as the implicit attribute in the sentiment control generation, and SENTIMENT = negative as the implicit attribute in the topic control generation. Prompts from Dathathri et al. (2020b) were used to generate continuation samples. We generated 20 samples for each attribute and prompt. More details are listed in Appendix A.1 and A.2. Table 2, in the single-attribute control tasks, Contrastive FPT achieves higher attribute relevance than prefix tuning-based baselines while having lower bias scores. This indicates that the generated texts are well controlled under the target explicit attribute without transferring by implicit attributes. In FPT, the perplexity score is the best among control-based baselines. The ablation experiment suggests that the proposed general prefix is essential for attribute control. Table 3 shows the generation samples of SENTI-MENT = positive from our models and baselines.

As shown in
In the FPT based model, there are more words with desired explicit attributes in generated texts, while there are more words with undesired explicit attributes contained in the baselines. More generation samples are given in Appendix B.

Models
In the multi-attribute control experiments, we added Distribution Lens (Gu et al., 2022) as a strong baseline. It searches for the intersection space of multiple attribute distributions as their combination for generating.

Experimental Settings
To explore the ability of FPT in the mult-attribute control task, we added a toxic comment dataset 3 for toxicity control. We used additional Google Per-

Model
Generated texts

GPT-2
The last time Dow and the SEC went shopping for a speed bump was Tuesday, in terms of ...

DExperts
The last time I saw Alvin Henderson, he said he hadn't done a rookie autograph. He says he hasn't played since...

Vanilla Prefix Tuning
The last time I saw this film was as a kid, I had to see it again for myself. There are...

Contrastive Prefix Tuning
The last time I saw the film, I didn't like it, and couldn't quite believe how much I ...

FPT
The last time I saw this film, it was a remarkable turning point in my career. It set the tone for the excellent...

Contrastive FPT
The last time I saw In the Hands of an Eagle was at this book release party. It was at a nice club...  spective API 4 to evaluate the relevance of toxicity.
Since it is meaningless to generate toxic content, so we only apply the non-toxic attribute in this task. We chose the first attribute as the topic attribute because we found that the filtered vocabulary size in logits manipulation of a topic attribute is larger than the other attributes (sentiment and nontoxic). The prompts used for generating samples are the same as in the sentiment control task. For each prompt, we generated 20 samples per attribute combination. More details are listed in Appendix A.3. Table 4 shows that our method can obtain comparable performance with the state-of-the-art approach. Distribution Lens, however, requires aggregating the datasets of all attributes to train its prefixes. If they hope to add a prefix to control a new attribute, 4 https://www.perspectiveapi.com/ they have to retrain all the prefixes. In contrast, FPT trains a prefix for each attribute individually and enables new attribute control prefixes to be added incrementally without retraining existing ones.

Conclusion
We proposed FPT, a prefix tuning-based method, to mitigate the effect of attribute transfer. FPT could encode implicit attributes in a dataset by a general prefix and use it to suppress the attribute transfer via inference-time logits manipulation. Results in the single-attribute control experiments showed that, with FPT, the generated texts can be more effectively controlled under the desired attribute with higher text fluency. Experimental results in the multi-attribute control suggested that FPT can achieve comparable performance to the state-ofthe-art approach while keeping the flexibility of adding new prefixes without retraining.
Although FPT shows better control ability, there are two points that need to be improved in the future. First, as in Gu et al. (2022), we need to select hyperparameter α to balance between the control ability and fluency in generated texts. Second, as shown in Table 5, although the time cost of FPT is lower than that of GeDi, it is higher than those of other prefix tuning-based methods and grows approximately linearly by a factor of 2 × number of attributes.

Model
Time (

A Experiment Setting Details
All the experiments are conducted on the basis of a GPT-2 Medium model. We freeze the parameters of the GPT-2 model when training all the prefixes. The length of all prefixes is set equal to 10. The GPU used for all training is a P40. For every task, we trained with 2 seeds and generate the texts.

A.1 Topic Control
Following the previous work (Qian et al., 2022), we use half of the data pieces in the AGNews dataset to obtain the general prefix and specific prefixes. The number of specific prefixes for this task is 4 (e.g. worlds, sports, business, and science). The other half of the dataset is used to train the classifier.
Following Gu et al. (2022), it is trained on the Deberta model (He et al., 2021), which is used to compute attribute relevance in this task. We train 8 epochs by setting the batch size to 8. We use AdamW as the optimizer and set the learning rate to 2e-5. To balance the performance between fluency and controllability, the hyperparameters α for generation are set to 1.1 and the top-p is set to 0.8. We spend 3.1 hours on 1 GPU to train the general prefix. The training time for each specific prefix is 1 hour for 1 GPU. The prompts for evaluation: "In summary,", "This essay discusses", "Views on", "The connection", "Foundational to this is", "To review", "In brief ", "An illustration of ", "Furthermore", "The central theme", "To conclude", "The key aspect", "Prior to this", "Emphasized are", "To summarize", "The relationship", "More importantly", "It has been shown", "The issue focused on", and "In this essay".

A.2 Sentiment Control
Following the previous work (Qian et al., 2022), we use half of the data pieces in the IMDb to get the general prefix and specific prefixes. The number of specific prefixes for this task is 2 (e.g. positive and negative). We set the batch size to 8, and the number of epochs to 50. We use AdamW as the optimizer, and the learning rate is set to 2e-5. To balance the performance between fluency and controllability, the hyperparameter α for generation is set to 3 and the top-p is set to 0.8. We spend 10 hours training general prefixes, and 5 hours for each specific prefix.

A.3 Multi-attribute Control
For the non-toxic attribute, we use 10,000 pieces of non-toxic labeled data to train the specific prefix. Then use another 20,000 pieces randomly sampled from the whole dataset to train the general prefix. In the multi-attribute control task, we set the batch size to 8, and the number of epochs to 8 for training the non-toxic specific prefix and general prefix. We use AdamW as the optimizer, and the learning rate is set to 2e-5. To balance the performance among attributes from different aspects, the combination of hyperparameters for generation is: To decide the first attribute, we choose 20 different prompts as input and obtain the filtered vocabulary sizes of different attributes. The average sizes of filtered vocabularies are shown in Table 7. We choose the attribute with the largest filtered vocabulary size in logits manipulation. When new attributes are added, this method can be used to decide the first attribute.
The prompts used for evaluation: "Once upon a time", "The book", "The chicken", "The city", "The country", "The horse", "The lake", "The last time", "The movie", "The painting", "The pizza", "The potato", "The president of the country", "The road", and "The year is 1910".

B Generated Samples
The more samples generated by our models and baselines are shown in Table 8 Table 7: Results of average filtered vocabulary size. We set all the α as 1.5. After filtering the vocabulary in logits manipulation, the specific prefix of the topic attribute guided model has the largest vocabulary size among these three attributes. We also found that the filtered vocabulary of the topic attribute can cover 85% of the filtered vocabulary of the sentiment attribute.

Model
Generated texts

GPT-2
The potato's ability to survive brings a new challenge to the traditional food truck love stage...

DExperts
The potato samples ranged in size from 0.6 mm to 5.1 mm in thickness. Analysis of proteins showing correlation with CSF CSF CSF...

Vanilla Prefix Tuning
The potato chip looks like a generic type of cheapo pin-up. It's supposed to be fun...

Contrastive Prefix Tuning
The potato chip's and biscuit's come up with the idea of making a film that is supposedly a true reflection of the experiences of students on campus...

FPT
The potato bomb! Potato bombs are one of the dumbest inventions ever. Their only purpose is to scare children....

Contrastive FPT
The potato crossing movie was stupid. Dumbly rushed and poorly acted. Dumb and poorly acted?...

Contrastive FPT
Prior to this season, we'd have heard rumours of an effort to rebuild the Knicks roster... Table 9: Samples generated by our models and baselines with the sports attribute. Desired explicit attribute: sports, undesired explicit attributes: world, business, science.

Generated texts
GPT-2 Emphasised are the events beyond the grave. The progenitor of darkness So I thought...

Vanilla Prefix Tuning
Emphasised are three key claims by Secretary of Defense Donald Rumsfeld on the war on terrorism....

Contrastive Prefix Tuning
Emphasised are odd and silly pension -and were he not so rich, he might have considered quitting politics...

FPT
Emphasised are the facts of the inner workings of the commodity markets and the profitability of global commodity trading...

Contrastive FPT
Emphasised are most oil-intensive', Australian manufacturing is the thirdmost-dependant on crude, official figures show... Table 10: Samples generated by our models and baselines with the business attribute. Desired explicit attribute: business, undesired explicit attributes: world, sports, science.

Model Generated texts
GPT-2 An illustration of the inner workings of the World Health Organization's Private Sector Vaccination Center...

Vanilla Prefix Tuning
An illustration of the Diamandis-Priest Fasting (2 cents) An illustration of the Diamandis-Priest Fasting...

Contrastive Prefix Tuning
An illustration of the biggest day in Spanish history in December 2017. Spanish government launches new campaign to promote ...

FPT
An illustration of the SBS / Getty Images virtual reality device at E3 last week. AP/E3Harms.com To catch up on the...

Contrastive FPT
An illustration of a proposed satellite CNET/Adrian Levy/UPI The most controversial satellite program in the past few years... Table 11: Samples generated by our models and baselines with the science attribute. Desired explicit attribute: science, undesired explicit attributes: world, sports, business.