Text-Transport: Toward Learning Causal Effects of Natural Language

As language technologies gain prominence in real-world settings, it is important to understand how changes to language affect reader perceptions. This can be formalized as the causal effect of varying a linguistic attribute (e.g., sentiment) on a reader's response to the text. In this paper, we introduce Text-Transport, a method for estimation of causal effects from natural language under any text distribution. Current approaches for valid causal effect estimation require strong assumptions about the data, meaning the data from which one can estimate valid causal effects often is not representative of the actual target domain of interest. To address this issue, we leverage the notion of distribution shift to describe an estimator that transports causal effects between domains, bypassing the need for strong assumptions in the target domain. We derive statistical guarantees on the uncertainty of this estimator, and we report empirical results and analyses that support the validity of Text-Transport across data settings. Finally, we use Text-Transport to study a realistic setting--hate speech on social media--in which causal effects do shift significantly between text domains, demonstrating the necessity of transport when conducting causal inference on natural language.


Introduction
What makes a comment on a social media site seem toxic or hateful (Mathew et al., 2021;Guest et al., 2021)?Could it be the use of profanity, or a lack of insight?What makes a product review more or less helpful to readers (Mudambi and Schuff, 2010;Pan and Zhang, 2011)?Is it the certainty of the review, or perhaps the presence of justifications?As language technologies are increasingly deployed in real-world settings, interpretability and explainability in natural language processing have become paramount (Rudin, 2019;Barredo Arrieta et al., 2020).Particularly desirable is the ability to understand how changes to language affect reader perceptions-formalized in statistical terms as the causal effect of varying a linguistic attribute on a reader's response to the text (Figure 1).
A core technical challenge for causal inference is that valid causal effects can only be estimated on data in which certain assumptions are upheld: namely, data where confounding is either fully measured or absent entirely (Rosenbaum and Rubin, 1983).Since confounding-the presence of factors that affect both the reader's choice of texts to read and the reader's response (e.g., political affiliation, age, mood)-is extremely difficult to measure fully in text settings, estimation of causal effects from natural language remains an open problem.One resource-intensive approach is to run a randomized experiment, which eliminates confounding by ensuring that respondents are randomly assigned texts to read (Holland, 1986;Fong and Grimmer, 2021).However, effects estimated from randomized experiments may not generalize outside of the specific data on which they were conducted (Tipton, 2014;Bareinboim and Pearl, 2021).Therefore, learning causal effects for a new target domain can require a new randomized experiment each time.
In this paper, we propose to bypass the need for strong data assumptions in the target domain by framing causal effect estimation as a distribution shift problem.We introduce TEXT-TRANSPORT, a method for learning text causal effects in any target domain, including those that do not necessarily fulfill the assumptions required for valid causal inference.Leveraging the notion of distribution shift, we define a causal estimator that transports a causal effect from a causally valid source domain (e.g., a randomized experiment) to the target domain, and we derive statistical guarantees for our proposed estimator that show that it has desirable properties that can be used to quantify its uncertainty.
We evaluate TEXT-TRANSPORT empirically using a benchmarking strategy that includes knowledge about causal effects in both the source domain and the target domain.We find that across three data settings of increasing complexity, TEXT-TRANSPORT estimates are consistent with effects directly estimated on the target domain, suggesting successful transport of effects.We further study a realistic setting-user responses to hateful content on social media sites-in which causal effects do change significantly between text domains, demonstrating the utility of transport when estimating causal effects from natural language.Finally, we conduct analyses that examine the intuition behind why TEXT-TRANSPORT is an effective estimator of text causal effects.

Problem Setting
Consider a collection of texts (e.g., documents, sentences, utterances) X , where person i (i = 1, . . ., N ) is shown a text X i from X .Using the potential outcomes framework (Neyman, 1923(Neyman, [1990]]; Rubin, 1974), let Y i (x) denote the potential response of respondent i if they were to read text x, where Y i : X → R. Without loss of generality, assume that each respondent in reality reads only one text X i , so their observed response is Y i (X i ).
Then the average response µ(P ) across the N respondents when texts X are drawn from a distri-bution P is given by: Now let X be parameterized as X = {a(X), a c (X)}, where a(X) is the text attribute of interest and a c (X) denotes all other attributes of the text X.Note that for a text causal effect to be meaningful, a(X) must be interpretable.This may be achieved by having a human code a(X) or using a lexicon or other automatic coding method.Again for simplicity, we assume a(X) ∈ {0, 1}.
Definition 1 (Natural causal effect of an attribute).Let P 1 (X) be a distribution such that a(X) = 1 and a c (X) ∼ P (a c (X)|a(X) = 1), and let P 0 (X) be a distribution such that a(X) = 0 and a c (X) ∼ P (a c (X)|a(X) = 0).
Then the causal effect of a(X) on Y is given by: Here, τ * is the natural effect of a(X).Linguistic attributes are subject to aliasing (Fong and Grimmer, 2021), in which some other linguistic attributes (e.g., the k-th linguistic attribute a c (X) k ) may be correlated with both the linguistic attribute of interest a(X) and the response Y , such that P (a c (X) k |a(X) = 1) ̸ = P (a c (X) k |a(X) = 0).For example, optimism may naturally co-occur with positive emotion, meaning that the natural effect of optimism also contains in part the effect of positive emotion.In contrast, the isolated effect would contain only the effect of optimism.In this paper, we choose to focus on natural effects due to the way linguistic attributes manifest in real-world language.That is, since optimism nearly always co-occurs with positive emotion, it may be difficult to interpret the effect of optimism when positive emotion is removed (the isolated effect), so we instead focus on their collective natural effect.
Our goal is then to learn τ T , the natural causal effect of the attribute a(X) on response Y in the text domain of interest P T .We consider use cases where it is not possible to directly estimate the effect from target data X ∼ P T , either because P T does not fulfill the assumptions required for valid causal inference or simply because the response Y is not measured in the domain of interest.

TEXT-TRANSPORT
To estimate causal effects under a target text distribution P T -without computing effects on P T directly-we propose TEXT-TRANSPORT, a method for transporting a causal effect from a source text distribution P R that does fulfill the assumptions required for valid causal inference and with respect to which P T is absolutely continuous.Our approach can help to generalize the causal findings of P R , which are specific to the data domain of P R , to any text distribution of interest P T .For mathematical convenience, we consider the source distribution P R to be a randomized experiment.We note that any crowdsourced dataset in which samples are randomly assigned to crowdworkers can be considered a randomized experiment.

Transporting effects
We characterize this problem as an instance of distribution shift, allowing us to define a causal effect estimator τ T that uses the density ratio between two distributions as an importance weight to transport the effect from P R to P T .Given X i ∼ P R , and letting dP T dP R (x) ≡ P T (x) P R (x) be the density ratio 1 between P T and P R , which gives us the effect estimate under P T : Intuitively, as all observed texts are drawn from P R , the role of the importance weight is to increase the contribution of texts that are similar to texts from P T , while reducing the contribution of texts that are representative of P R .That is, if P T (X i ) is high and P R (X i ) is low, then X i will have a greater contribution to μ(P T ).To highlight this transport of P R to P T , in the remainder of this paper we will refer to μ(P T ) as μR→T .
A strength of this estimator is that we are able to quantify statistical uncertainty around the causal effect.We demonstrate (with derivations and proofs in Appendix B) that μR→T has a number of desirable properties that allow us to compute statistically valid confidence intervals: (1) it is an unbiased estimator of µ(P T ), (2) it is asymptotically normal, and (3) it has a closed-form variance and an unbiased, easy-to-compute variance estimator.
1 More generally, this is the Radon-Nikodym derivative, formally defined in Appendix A.

Importance weight estimation
Estimating the transported response μ(P T ) first requires computing either the derivative dP T dP R (X) or the individual probabilities P R (X), P T (X).While there are many potential ways to estimate this quantity, we propose the classification approach TEXT-TRANSPORT clf and the language model approach TEXT-TRANSPORT LM (Figure 2).
TEXT-TRANSPORT clf .The classification approach for estimating dP T dP R (X) relies on the notion that the density ratio can be rewritten in a way that makes estimation more tractable.Let C denote the distribution (or corpus) from which a text is drawn, where C = T denotes that it is drawn from P T and C = R denotes that it is drawn from P R .Then dP T dP R (X) can be rewritten as follows: P (C = T |X) and P (C = R|X) can be estimated by training a binary classifier M θ : X → {0, 1} to predict if a text X came from T or R. P (C = R) and P (C = T ) are defined by their sample proportions (i.e., by their proportion of the total text after combining the two corpora). 2EXT-TRANSPORT LM .
Because language models are capable of learning text distributions, we are able to take an alternative estimation approach that does not require learning dP T dP R (X).A language model trained on samples from P R or P T , for instance, can compute the probability of texts under the learned distributions.
Pre-trained large language models (LLMs) are particularly useful for estimating P R and P T , since their training corpora are large enough to approximate the distribution of the English language, and their training data is likely to have included many P R or P T of interest.Following recent advances in LLMs, one way of obtaining P R (X) and P T (X) from an LLM is to prompt the LLM in a way that induces it to focus on P R or P T .Once the LLM has been prompted toward either P R or P T , sentence probabilities from the LLM can be treated as reflections of P R (X) or P T (X), respectively.We provide examples of such prompts in Figure 2, and we explore this approach in our experiments.

Experimental Setup
We conduct empirical evaluations to assess the validity of causal effects obtained through TEXT-TRANSPORT in three different data settings of increasing complexity (Table 1).

Evaluation methodology
To assess the validity of TEXT-TRANSPORT, we conduct experiments comparing the transported average response μR→T with μR , the average response under P R , and μT , the average response under P T .A valid transported response μR→T will be similar to μT and significantly different from μR .To quantify these differences, we compare estimated averages and 95% confidence intervals, as well as normalized RMSE (RMSE divided by the standard deviation of the target response) between μR→T and μT .
As we mention previously, validating transported causal effects requires an evaluation strategy in which causal effects under P R and P T are both known.Therefore, each of our evaluation datasets consists of a single crowdsourced dataset that can be divided into two parts (e.g., a corpus of movie reviews can be split by genre), such that they possess the following properties.First, to allow μR and μT to be computed directly, both P R and P T are randomized experiments (we reiterate that P T is a randomized experiment only for the purposes of validation; we would not expect actual P T s to be randomized).Second, the response Y is measured for both P R and P T .Third, P R and P T are distinct in a way where we would expect the average response to differ between the two.This allows us to evaluate whether transport has successfully occurred (if the average response is the same between P R and P T , transport will not do anything).We choose three crowdsourced datasets, partition each into P R and P T , and compute μR , μT , and μR→T .We estimate confidence intervals for each average response through bootstrap resampling with 100 iterations.
Baselines.While a small number of prior studies have proposed estimators of some type of text causal effect from observational (i.e., nonrandomized) data, effects obtained from these methods are not directly comparable to those obtained using TEXT-TRANSPORT (further discussion of these methods can be found in Section 6).However, rather than using the density ratio to transport effects, other transformations of the source distribution to the target distribution are possible.One intuitive baseline is to train a predictive model on the source distribution, which is then used to generate pseudo-labels on the target distribution.These pseudo-labels can be averaged to produce a naive estimate (the naive baseline).

Datasets
Amazon (controlled setting).The Amazon dataset (McAuley and Leskovec, 2013) consists of text reviews from the Amazon e-commerce website, where the reader response is the number of "helpful" votes the review has received.We choose reviews of musical instruments as P R and reviews of office supplies as P T .
To construct a best-case scenario in which there are no unmeasured factors in the data, we generate a new semi-synthetic response Y by predicting the number of helpful votes as a function of a(X), a c (X).We use a noisy version of this prediction as our new Y .This ensures that all predictable variability in the response Y is captured in the text.Furthermore, we sample reviews into P R and P T according to their predicted likelihood of being in P R or P T when accounting only for a(X), a c (X).This provides a controlled data setting in which we know that a model is capable of distinguishing between P R and P T , such that we can evaluate TEXT-TRANSPORT under best-case conditions.
EmoBank (partially controlled setting).The EmoBank dataset (Buechel and Hahn, 2017) consists of sentences that have been rated by crowdworkers for their perceived valence Y (i.e., the positivity or negativity of the text).To construct P R , we sample sentences such that texts with high writer-intended valence occur with high probability, and to construct P T , we sample sentences such that texts with low writer-intended valence occur with high probability.This partially controls the data setting by guaranteeing that the source and target domains differ on a single attribute-writerintended valence-that is known to us (but hidden from the models).
Hate Speech (natural setting).The Hate Speech dataset (Qian et al., 2019) consists of comments from the social media sites Reddit and Gab.These comments have been annotated by crowdworkers as hate speech or not hate speech.The Reddit comments are chosen from subreddits where hate speech is more common, and Gab is a platform where users sometimes migrate after being blocked from other social media sites.To represent a realistic data setting in which we have no control over the construction of the source and target distributions, we treat the corpus of Reddit comments as P R and the corpus of Gab comments as P T .

Implementation
Linguistic attributes.To automatically obtain linguistic attributes a(X), a c (X) from the text, we use the 2015 version of the lexicon Linguistic Inquiry and Word Count (LIWC) to encode the text as lexical categories (Pennebaker et al., 2015).LIWC is a human expert-constructed lexicongenerally viewed as a gold standard for lexiconswith a vocabulary of 6,548 words that belong to one or more of its 85 categories, most of which are related to psychology and social interaction.We binarize the category encodings to take the value 1 if the category is present in the text and 0 otherwise.
TEXT-TRANSPORT clf .We use the following procedure to implement TEXT-TRANSPORT clf .First, we consider data D R from P R and data D T from P T .We take 10% of D R and 10% of D T as our classifier training set D train .Next, we train a classifier M θ on D train to distinguish between P R and P T .For our classifier, we use embeddings from pre-trained MPNet (Song et al., 2020), a wellperforming sentence transformer architecture, as inputs to a logistic regression.
From M θ , we can obtain P (C = T |X) and P (C = R|X) for all texts X in the remaining 90% of D R .We compute P (C = R) and P (C = T ) as In the case of the Amazon dataset, we note that although we can estimate the classification probabilities as P (C = T |X), P (C = R|X), the true probabilities are already known, as we use them to separate texts into P R and P T .Therefore-to evaluate the effectiveness of TEXT-TRANSPORT under conditions where we know the classifier to be correct-we use the known probabilities P (C = T |X), P (C = R|X) in our evaluations on the Amazon dataset only.
TEXT-TRANSPORT LM .This approach can be implemented without any training data, leaving the full body of text available for estimation.In our experiments, we estimate P R (X) and P T (X) through prompting.For each dataset, we provide pre-trained GPT-3 with a prompt that describes P R or P T , then follow the prompt with the full text from each sample X ∼ P R .On the EmoBank dataset, for instance, we provide GPT-3 with the prompts "You are writing a positive statement" (for P R , the high-valence distribution) and "You are  writing a negative statement" (for P T , the lowvalence distribution).A full list of prompts can be found in Appendix D.3.After prompting the model with either P R or P T , we compute the token probabilities over each X, then compute sentence probabilities as the product of the token probabilities.If a text has multiple sentences, we treat the average of the sentence probabilities as the overall text probability.Finally, we compute the ratio as our importance weight.

Validity of TEXT-TRANSPORT
We evaluate the validity of our TEXT-TRANSPORT responses on the Amazon, EmoBank, and Hate Speech data settings (Figure 3, Table 2).
We observe that on the Amazon dataset, both TEXT-TRANSPORT clf and TEXT-TRANSPORT LM are well-validated.For both sets of Amazon re-sults, our transported response μR→T is statistically significantly different from μR , while being statistically indistinguishable from μT .In contrast, the naive baseline produces an estimate with confidence intervals that overlap both μR and μT , and its RMSE is higher than both TEXT-TRANSPORT estimates.The success of TEXT-TRANSPORT clf in this setting suggests that if the classifier is known to be a good estimator of the probabilities P (C = T |X) and P (C = R|X), the transported estimates will be correct.The success of TEXT-TRANSPORT LM in this setting, on the other hand, suggests that prompting GPT-3 can in fact be an effective way of estimating P R (X) and P T (X) and that the ratio between the two can also be used to produce valid transported estimates.
We further find that as the data setting becomes less controlled (i.e., EmoBank and Hate Speech), our transported responses continue to show encouraging trends-that is, the transported effect indeed moves the responses away from μR and toward μT , while transported responses from the naive baseline exhibit little to no movement toward the target.When evaluating TEXT-TRANSPORT LM on EmoBank, μR→T and μT have no statistically significant difference.However, in evaluations of TEXT-TRANSPORT clf , we find that μR→Tthough transported in the right direction-retains a statistically significant difference from μT ; and when evaluating TEXT-TRANSPORT LM on Hate Speech, we observe wide confidence intervals for μR→T that cover both μT and μR , though the point estimates of μR→T and μT are very close.
Finally, we note that TEXT-TRANSPORT LM is less stable than TEXT-TRANSPORT clf with respect to the width of its confidence intervals, although the transported point estimates are better.This is particularly highlighted by the higher RMSE of TEXT-TRANSPORT LM compared to the naive baseline on the Hate Speech dataset, in spite of TEXT-TRANSPORT LM 's much better point estimate. 3  We posit that the instability of TEXT-TRANSPORT LM is due to the very small probability of any particular text occurring under a given probability distribution, as well as a potential lack of precision introduced when using prompting to target an LLM to a specific distribution.We observe that both P R (X) and P T (X) are typically both very small, and any difference between them-while minute in absolute terms-is amplified when taking their ratio.As a result, the range of importance weights under TEXT-TRANSPORT LM is much larger than the range of dP T dP R (X) under TEXT-TRANSPORT clf , introducing a large amount of variability when estimating μR→T .
Often, however, TEXT-TRANSPORT LM can still produce reasonable confidence intervals (as is the case for the Amazon and EmoBank datasets), and it illustrates the efficacy of the TEXT-TRANSPORT method under one of the simplest implementations (since no additional model training or even access to target data is required).
3 In this sense, metrics like RMSE can be somewhat reductive, as they penalize the larger confidence interval but fail to capture the fact that the "transported" effect under the naive estimator has moved very little toward the target distribution, while the transported effect under the LM approach has correctly made a much larger shift toward the target distribution.In this section, we highlight a realistic use case of TEXT-TRANSPORT.Taking our Hate Speech dataset, we examine the source, transported, and target effects of five linguistic attributes, where Reddit is again the source distribution and Gab is the target distribution (Figure 4).Transported effects are estimated using TEXT-TRANSPORT clf .

A realistic use case: What makes a comment hateful?
We find that the causal effects of these five linguistic attributes estimated directly on Reddit differ significantly from their counterparts that have been transported to Gab.Though we would not have access to effects estimated directly on the target distribution in a real-world setting, we are able to validate here that the effect shifts are consistent with causal effects directly estimated from Gab.
Negative causal effects.After transport to Gab, the linguistic attributes netspeak, insight, and family are all shown to have significant negative effects on whether a comment is perceived as hate speech, while they are found to have no significant effect in the original Reddit data.In other words, when Gab users use netspeak, make insights, or talk about family, these conversational choices cause readers to perceive the overall comment as less hateful, while the same does not hold true for Reddit users.
Positive causal effects.In contrast, after transport to Gab, the linguistic attribute function is shown to have a significant positive effect on whether a comment is perceived as hate speech, while it was found to have no significant effect in the original Reddit data.Function words comprise articles, pronouns, conjunctions, negations,

Language modeling approach
They have the financial backing of powerful special interests and the determination to ignore the will of the people of Alaska and the American public.
Pushed as an effort to promote "Sonoma County" wines and a consumer education effort, the new law instead forces vintners to needlessly sully their package and undermines their own marketing efforts.
The efforts to save the mills failed partly because the international market for steel had largely dried up.
Yes, Self, I am also bothered that this observation ignores half-eaten cheese sandwiches, incomplete insect collections, and locks of infants' hair, forgotten in closets, basements, and warehouses.
Opera Cancelled Over a Depiction of Muhammad Where is the authority?
7 Die As US Helicopter Crashes in Iraq Suddenly, Amy screamed.
Table 3: Texts from the EmoBank source distribution P R (texts with high writer-intended valence) with the highest TEXT-TRANSPORT importance weights.Target distribution P T comprises texts with low writer-intended valence.
and other words that serve primarily grammatical purposes, and prior work has found that they can be highly suggestive of a person's psychological state (Groom and Pennebaker, 2002;Chung and Pennebaker, 2007;Pennebaker, 2011).
Though the difference between the original and transported effect is not statistically significant, profanity is also found to have a more positive effect on whether a comment is perceived as hate speech after transport to Gab compared to Reddit.This indicates that Gab users' use of profanity in their comments causes readers to perceive the overall comment as more hateful than if a Reddit user were to make profane remarks.This effect shift may be explained by the specific nature of the profanity used on Gab, which is qualitatively harsher and more offensive than the profanity used on Reddit.
The differences in these transported and original causal effects emphasize the importance of our method.An automatic content moderation algorithm, for instance, would likely need to consider different linguistic factors when deciding which comments to flag on each site.

An intuition for effect transport
Previously in Section 3.1, we stated that the intuition behind the change of measure dP T dP R (X) as an importance weight in the estimator μR→T was to increase the contribution of texts that are similar to P T , while reducing the weight of texts that are most representative of P R .To explore whether this is indeed the case, we identify the texts from EmoBank with the highest importance weights for each of our estimation approaches (Table 3).Texts with large importance weights have high P T (X) and low P R (X), meaning they should be similar to texts from P T despite actually coming from P R .
We observe that texts from P R (i.e., texts with greater probability of high writer-intended valence) with high weights are in fact qualitatively similar to texts from P T (i.e., texts with greater probability of low writer-intended valence).That is, although texts from P R should be generally positive, the texts with the highest weights are markedly negative in tone, making them much more similar to texts from P T .These observations support the intuition that the change of measure transports causal effects to the target domain by looking primarily at the responses to texts in the source domain that are most similar to texts in the target domain.

Related Work
TEXT-TRANSPORT draws on a diverse body of prior work from machine learning and causal inference, including the literature on domain adaptation, distribution shift, and generalizability and transportability.We build on methods from these fields to define a novel, flexible estimator of causal effects in natural language.
Text causal effects.A number of approaches have been proposed for estimating causal effects from natural language.Egami et al. (2018) construct a conceptual framework for causal inference with text that relies on specific data splitting strategies, while Pryzant et al. (2018) describe a procedure for learning words that are most predictive of a desired response.However, the interpretability of the learned effects from these works is limited.In a subsequent paper, Pryzant et al. (2021) introduce a method for estimating effects of linguistic proper-ties from observational data.While this approach targets isolated effects of linguistic properties, it requires responses to be measured on the target domain, and it accounts only for the portion of confounding that is contained with the text.
Finally, in a recent paper, Fong and Grimmer (2021) conduct randomized experiments over text attributes to determine their effects.While allowing for valid causal inference, the resulting constructed texts are artificial in nature and constitute a clear use case for TEXT-TRANSPORT, which can transport effects from the less-realistic experimental text domain to more naturalistic target domains.
In parallel, a line of work studying the external validity of estimated causal effects has emerged within statistical causal inference (Egami and Hartman, 2022;Pearl and Bareinboim, 2022).These works aim to understand the conditions under which causal effects estimated on specific data can generalize or be transported to broader settings (Tipton, 2014;Bareinboim and Pearl, 2021).Prior work has also used density ratio-style importance weights to estimate average causal effects with high-dimensional interventions (de la Cuesta et al., 2022;Papadogeorgou et al., 2022).
We emphasize that TEXT-TRANSPORT is conceptually novel and methodologically distinct from these prior works.In our work, we explore an open problem-causal effect estimation from text-and define a new framework for learning causal effects from natural language.We propose a novel solution that uses tools from the domain adaptation and transportability literature, and we introduce novel methods for estimating natural effects in practice in language settings.

Conclusion
In this paper, we study the problem of causal effect estimation from text, which is challenging due to the highly confounded nature of natural language.To address this challenge, we propose TEXT-TRANSPORT, a novel method for estimating text causal effects from distributions that may not fulfill the assumptions required for valid causal inference.We conduct empirical evaluations that support the validity of causal effects estimated with TEXT-TRANSPORT, and we examine a realistic data setting-hate speech and toxicity on social media sites-in which TEXT-TRANSPORT identifies significant shifts in causal effects between text domains.Our results reinforce the need to account for distribution shift when estimating text-based causal effects and suggest that TEXT-TRANSPORT is a compelling approach for doing so.These promising initial findings open the door to future exploration of causal effects from complex unstructured data like language, images, and multimodal data.

Acknowledgements
This material is based upon work partially supported by the National Science Foundation (awards 1722822 and 1750439) and the National Institutes of Health (awards R01MH125740, R01MH132225, R01MH096951, and R21MH130767).Victoria Lin is partially supported by a Meta Research PhD Fellowship.Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.

Limitations
Although TEXT-TRANSPORT is effective in accounting for distribution shift to estimate effects of linguistic attributes in target domains without data assumptions, the method relies on the existence of a source domain that satisfies the data assumptions required for valid causal inference.Such a source domain-even a small or limited one-may not always be available.We plan to address this limitation in future work, where transport from any source domain is possible.
Additionally, TEXT-TRANSPORT proposes a framework for estimating natural causal effects from text.However, as we discuss above, in some cases it may also be of interest to estimate isolated causal effects from text.In future work, we will extend TEXT-TRANSPORT to include an estimator for isolated causal effects.
Finally, a requirement of the target distribution P T is that it is absolutely continuous with respect to the source distribution P R .The absolute continuity assumption is violated if a text that would never occur in P R could possibly occur in P T .Therefore, this assumption may not be satisfied if the source and target distributions are completely unrelated and non-overlapping, even in latent space.Practically speaking, this means that it may not be possible to transport effects between distributions that are extremely different: for instance, from a corpus of technical manuals to a corpus of Shakespearean poetry.
10 Ethics Statement Broader impact.Language technologies are assuming an increasingly prominent role in realworld settings, seeing use in applications like healthcare (Wen et al., 2019;Zhou et al., 2022;Reeves et al., 2021), content moderation (Pavlopoulos et al., 2017;Gorwa et al., 2020), and marketing (Kang et al., 2020).As these black-box systems become more widespread, interpretability and explainability in NLP are of ever greater importance.TEXT-TRANSPORT builds toward this goal by providing a framework for estimating the causal effects of linguistic attributes on readers' responses to the text.These causal effects provide clear insight into how changes to language affect the perceptions of readers-an important factor when considering the texts that NLP systems consume or produce.
Ethical considerations.TEXT-TRANSPORT relies on pre-trained large language models to compute text probabilities.Consequently, it is possible that these text probabilities-which are used to transport causal effect estimates-may encode some of the biases contained in large pre-trained models and their training data.Interpretations of causal effects produced by TEXT-TRANSPORT should take these biases into consideration.
Additionally, we acknowledge the environmental impact of large language models, which are used in this work.
Putting the two cases together,

Figure 1 :
Figure 1: TEXT-TRANSPORT facilitates estimation of text causal effects on any target domain by transporting causal effects from a source domain.

Figure 2 :
Figure 2: Two proposed approaches for estimating importance weights.

1
|D R | and 1 |D T | , respectively.Then we have dP R (X) dP T (X) = P (C = T |X)

Figure 3 :
Figure 3: Validation of transported responses μR→T against known target responses μT and source responses μR .

Figure 4 :
Figure 4: Source and transported natural causal effects from Reddit to Gab.

Table 1 :
Source (P R ) and target (P T ) distributions for each evaluation dataset.

Table 2 :
Normalized RMSE of transported responses μR→T against known target responses μT .