Exploring Self-Identified Counseling Expertise in Online Support Forums

A growing number of people engage in online health forums, making it important to understand the quality of the advice they receive. In this paper, we explore the role of expertise in responses provided to help-seeking posts regarding mental health. We study the differences between (1) interactions with peers; and (2) interactions with self-identified mental health professionals. First, we show that a classifier can distinguish between these two groups, indicating that their language use does in fact differ. To understand this difference, we perform several analyses addressing engagement aspects, including whether their comments engage the support-seeker further as well as linguistic aspects, such as dominant language and linguistic style matching. Our work contributes toward the developing efforts of understanding how health experts engage with health information- and support-seekers in social networks. More broadly, it is a step toward a deeper understanding of the styles of interactions that cultivate supportive engagement in online communities.


Introduction
Online social media forums play a critical role in health-related information sharing (Record et al., 2018). Health experts have noted that they can help reduce healthcare inequalities and improve access to health care, for instance by empowering coalitions of people living with chronic illness or specific disabilities (Griffiths et al., 2012), or by providing an anonymous forum for people seeking emotional support (De Choudhury and De, 2014). On the other hand, these forums elevate concerns about spreading medically inaccurate, misleading, or unsound information (Domínguez and Sapiña, 2015;Gage-Bouchard et al., 2018), which has had harmful public health impacts (Poland et al., 2011;Nobles et al., 2019). One study concluded that health information seekers in forums such as Reddit are likely to enact suggested behaviors regardless of perceived credibility (Record et al., 2018). However, the researchers also noted that this openness to information could be an opportunity for experts to encourage healthy behaviors through information sharing.
In this landscape, it is critical to understand the dynamics that cultivate safe communities that benefit the health and well-being of their participants and the broader implications for health communication (Chou et al., 2009). Health experts are thus considering social media's role in their interactions with patients and broader public health issues, and their role in engaging with the platforms (Domínguez and Sapiña, 2015;Nobles et al., 2019Nobles et al., , 2020. This motivates an important research direction: understanding how experts engage with users in online platforms. This can inform platform design, moderation decisions, and health promotion efforts by experts. This work focuses on understanding the engagement with professionals in the domain of mental health with two main research questions: (RQ1) Do experts have distinct influences as compared to non-experts in their interactions with supportseekers in online mental health?; and (RQ2) Do the experts' behaviors reflect established counseling principles and findings regarding behaviors associated with positive counseling outcomes? To answer these questions, we analyze responses from self-identified mental health professionals (MHP) to support-seekers in mental health and support communities on Reddit, and compare them to responses from other users who we refer to as peers. This is an important comparison, as many peers share similar health experiences, which prior work has found is associated with higher empathic concern (Hodges et al., 2010).
First, we test whether a text classifier can distin-guish between responses to support seekers from MHPs and peers. We find that it can, with 70% accuracy (well above random chance of 50%). Second, we analyze comments leading to further engagement with the support-seeking posters, as existing counseling principles emphasize the importance of eliciting client engagement in expert counseling sessions (Miller and Rollnick, 2012;Pérez-Rosas et al., 2018). Third, we analyze the users' linguistic tendencies, drawing inspiration from analyses of counseling conversations, which have offered insight into counselor behaviors associated with high quality sessions grounded in existing theories from psychology and counseling research using computational methods (Althoff et al., 2016;Pérez-Rosas et al., 2018;Zhang et al., 2019;Miller and Rollnick, 2012).
The main contributions of this work are: (1) We construct a dataset of mental health conversations from Reddit users with self-identified counseling expertise, covering a set of mental health subreddits annotated with categories denoting the type of mental health concern; (2) We develop a classifier that can distinguish between the language of MHPs and that of peers; (3) We perform an analysis of the differences in language use between MHPs and peers; and (4) We provide insight into language that leads to further engagement with support-seekers, comparing responses to peers and MHPs.

Related Work
Studies within the education and health domains have shown that advice and help-seeking interactions in online communities contribute positively to users ' well-being, learning, and skills development (Campbell et al., 2016;Wang et al., 2015). This is particularly true for applications such as computer programming, career development, mentoring, coping with chronic or life-threatening diseases, and mental health issues (Baltadzhieva and Chrupała, 2015;Tomprou et al., 2019;Wang et al., 2015;De Choudhury and De, 2014).
In the mental health domain, studies have explored online support communities and many have found positive outcomes associated with anonymity, perceived empathy, and active user engagement (De Choudhury and De, 2014;Rheingold, 1993;Hodges et al., 2010;Welbourne et al., 2009;Nambisan, 2011). Computational approaches have aided studies in mental health forums, helping reveal positive relationships between linguistic accommodation and social support across subreddits (Sharma and De Choudhury, 2018). One example of insights from this work is that topicfocused communities like subreddits may enable more peer-engagement than non-community based platforms (Sharma et al., 2020). Other studies have revealed certain trade-offs of online support platforms, such as disparities in the level of support offered toward support-seekers of various demographics (Wang and Jurgens, 2018;Nobles et al., 2020) and in condolences extended across different topics of distress (Zhou and Jurgens, 2020). Studying MHP behaviors in such scenarios might help develop approaches that balance these trade-offs.
Computational approaches applied in these forums have also shed light on population-level health trends and health information needs, with examinations into how depression and post-traumatic stress disorder (PTSD) affect different demographic strata (Amir et al., 2019). Data mining has also been applied to understand adverse drug reactions (Wang et al., 2014) and public reactions towards infectious diseases (Park and Conway, 2017). Nobles et al. (2018) highlighted the potential for these forums to aid targeted health communication, for example by sharing information in r/STD, a subreddit about sexually transmitted diseases. Another case study of r/STD revealed the prevalence of diagnoses requests, and suggested that health professionals could partner with social media platforms to positively influence crowd-sourced diagnoses and help mitigate harmful misdiagnoses (Nobles et al., 2019). Record et al. (2018) found that health information seeking Reddit users are likely to enact suggested behaviors regardless of perceived credibility, providing further reason for health expert engagement to intervene when harmful information sharing occurs and promote healthy behavior.
Fewer studies have analyzed expert interactions in online forums. A study in a large Q&A community found that experts are more likely to provide help than peers and that their participation in discussions resulted in increased length and substance of discussions (Procaci et al., 2017). Recent studies have compared interactions with experts to interactions with peers in broader scientific communities (Park et al., 2020) and r/AskDocs on Reddit (Nobles et al., 2020). The latter paper closely relates to our study, as they also consider posts from experts on Reddit, but solely within r/AskDocs about different health topics and with   users of varying demographics. The insights discussed above motivate investigations into how health experts and other users promote scientifically sound advice and offer supportive responses to health information seekers in online forums. In this work, we aim to contribute additional insights into expertise influence in online mental health communities by studying the dynamics of the communication process between support seekers and support providers.

Data Collection
We seek to understand the tendencies of users with professional experience, and more specifically counseling expertise, when interacting with support-seekers in online mental health and support-related forums. In uncovering which tendencies are associated with expertise, we enable further investigation into their role in the social dynamics of online support-seeking interactions, and potential applications of insight-driven recommendations for moderators and users of these forums.
Source. We use Reddit for its quantity of publicly available interactions in communities called subreddits that discuss mental health issues. In addition, Reddit has a system that allows users to indicate their professional expertise (Reddit Flairs), which we use to identify a set of users with mental health professional background, identified as MHPs during our study. We obtained flairs from the r/psychotherapy subreddit, 1 a decision motivated by their reliability, as the moderators of this community allow comments and posts only by licensed therapy providers who may be asked to submit proof if concerns of falsely posing as a therapist arise. 2 Sample flair tags in this set are: Psychiatrist (sometimes accompanied by MD or DO), LPC (or Licensed Professional Counselor), LMFT (or Licensed Marriage and Family Therapist), PsyD (or Doctorate of Psychology).
We use an existing list of mental health subreddits from r/ListOfSubreddits 3 with additions from manual observations; all of the subreddits in our dataset with their number of comments are in Appendix B in Table 5. From these, we retrieve threads where an MHP submitted a direct reply. During this step, we also kept posts made by peers i.e., individuals who did not use any of the mental health care professional flairs. Our collection spans threads created between November 29, 2009 and December 21, 2020. Table 1 shows descriptive statistics for the final composition of the dataset, and Table 2 shows a sample interaction demonstrating the structure we use for our analysis. This study focuses on direct replies to the poster, thus we attempt to eliminate megathreads which tend not to focus in individual support-seekers by removing those above the 95th percentile in their number of direct replies; we leave analysis of deeper nested replies for future work.
Health Topics. To understand whether particular topics influence interactions with support-seekers, we group the subreddits into broader topics based h e a l t h t e n t a t w o r k a n   Figure 1: LIWC category dominance scores, computed as the relative use by peers divided by the relative use by MHPs, so that equal use is at y = 1 (blue line), higher dominance by peers at y > 1 (grey bars) and higher dominance by MHPs at y < 1 (white bars). Showing categories where frequency of use differs by at least 10%.  on related health domains. We begin by following the categorization of subreddits by Sharma and De Choudhury (2018), who used the k-means clustering algorithm to generate initial clusters on the n-grams (n = 3) of the posts and manually refined the categories based the community descriptions in their subreddit home pages. Next, we adjust the categories and their associated subreddits based on the World Health Organization's ICD-10 classification system of mental and behavioural disorders. 4 The resulting topic categories are listed in Table 3 alongside shortened names which we use to refer to them. The full list of subreddits assigned to each topic are listed in Appendix B in Table 6.

Distinguishing MHPs and Peers
To begin our investigation into the linguistic behaviors of MHPs and peers, we test whether simple text classifiers are able to distinguish between comments authored by either MHPs or peers. We build three classifiers with different feature sets; the first are unigram counts for unigrams occurring at least five times, the second includes counts for the 73 word classes in the LIWC (Linguistic Inquire and Word Count) lexicon (Pennebaker et al., 2015), and the third encodes a subset of LIWC word classes associated with perspective shifts (i.e., focusfuture, focuspast, focuspresent, I, ipron, negemo, posemo, ppron, pronoun, shehe, they, we, and you) (Althoff et al., 2016); we elaborate on the psychological meaning behind these features in our analyses in the next section. Due to the class imbalance between the peer and MHPs classes, we first downsampled the peer class to get a balanced distribution with the MHP class. This resulted in a set of 9,685 instances per class. We conduct our evaluations using ten-fold cross validation. Across these folds, the number of features ranges from 8,668 to 8,703. We use a Naive Bayes model, implemented with Sklearn's Multino-mialNB module, 5 which outperformed a logistic regression model and an SVM in preliminary experiments. 6 All models outperform a random baseline 7 with all LIWC features bringing the accuracy to 59.12%, LIWC perspective features to 59.14%, and unigram features to 70.80%. Overall, the classification results indicate language differences exist between the MHPs and peers. Motivated by this result, we proceed to several analyses to gain insights.

Linguistic and Dialogue Analysis
We analyze the linguistic behaviors of MHPs and peers responding to support-seeking posts, and their potential influence in eliciting further engagement with the support-seeker. Our analyses are inspired by psychology and computational studies that have shown that conversational behavioral aspects such as word usage, client engagement, and 5 https://scikit-learn.org/stable/ modules/generated/sklearn.naive_bayes. MultinomialNB.html 6 Runs in ∼40 seconds per fold on one AMD Ryzen 7 3700U CPU.

Linguistic Ethnography
Numerous studies have demonstrated relationships between the dominant usage of certain word categories with individuals' psychological and physical health (Tausczik and Pennebaker, 2010;Weintraub, 1981;Rude et al., 2004). In alignment with these studies, we investigate the usage of such word categories using the LIWC and WordNet-Affect lexicons (Pennebaker et al., 2015;Strapparava and Valitutti, 2004).
For each group of users, we first compute the proportion of their words that fall in each category. Then, we compute the dominant use by dividing the proportion for peer users over the proportion for MHPs (Mihalcea and Pulman, 2009). Figure 1 shows LIWC categories where the rate of use differs by at least 10%, and results for WordNet Affect categories are shown in Figure 2.
Some observations such as the higher dominance of swear words (swear) and internet speak (netspeak) might be expected if professionals avoid such language. An interesting contrast in peers' language is the dominant use of first-person pronouns (I, we) and focus on the past (focuspast). In contrast, MHPs seem to use more non-first person pronouns (you, they) and focus on the future (focusfuture) instead. Peers' use of first person pronouns might arise when they share similar experiences with support-seekers. MHPs' use of second-person pronouns might suggest they are focusing on the support-seekers' experiences as a counselor would with a client in a counseling encounter. We also observe higher dominance of all WordNet Affect categories among peers, however the joy category (the most positive), is nearly equal with MHPs.
These observations of the peers' language are compelling because they align with existing theories linking depression to negative views of the future (i.e., focuspast and negative WordNet af-fects) (Pyszczynski et al., 1987) and self-focusing style (i.e., first-person pronouns) (Pyszczynski and Greenberg, 1987;Campbell and Pennebaker, 2003). Likewise, clients of SMS-based crisis counseling conversations were more likely to report feeling better after the encounter if they exhibited perspective shifts from these categories to their counterparts (i.e., toward focusfuture, non-first person pronouns, and positive sentiment) (Althoff et al., 2016).
Interestingly, the same study found clients were more likely to shift perspective when their counselors exhibited use of the counterpart categories first, suggesting that the counselors may play a key role in helping drive the perspective shift. Given those positive outcomes, observing the same dominant linguistic aspects among MHPs is encouraging and potentially signals a connection between how counselors apply conversational behaviors in practice and in online forum interactions. Future work can investigate the progression of dialogue between MHPs and support-seekers to find if supportseekers similarly exhibit the perspective shifts associated with the positive outcomes of the prior study, and likewise whether users of the forums also experience positive outcomes where this occurs.

Engaging Support-Seekers
To understand if linguistic behaviors are associated with prompting further engagement with the support-seeker, we compare the dominance of LIWC categories in comments receiving replies compared to comments that do not by dividing the usage rates of the former by the latter. Figure 3 shows these ratios for categories that differ by at least 5%. A compelling observation is the dominance of the categories health, tentat, and you in the MHP comments prompting poster-replies, and you, focusfuture, interrog, and health in the peer comments prompting poster-replies, as was exhibited among MHPs (see Figure 1 in Section 5.1); on the other end, the categories are more dominant in comments that do not engage a reply such as I, we, death, friend, relig, swear, were similarly represented as dominant categories in the peer group.
To gain further insight into these observations, we perform the following analysis: for each user group (peers and MHPs) we use a foreground corpus of their comments that were replied to by the support-seekers, and a background corpus of their comments that were not, and compute the dominance of LIWC categories of the foreground over  Figure 3: Dominance of LIWC categories, computed as the category relative frequencies among comments that prompt support-seeker responses divided by the relative frequencies among comments that do not, computed separately for MHPs (left) and peers (right).  the background as a ratio of their relative frequencies. We then rank the categories by highest to lowest dominance scores, and refer to this ranking by DRR (for Dominance Rank for Replied comments). We compare the DRRs of each user group to the ranking of LIWC category usage among MHP users and among peer users separately (from Section 5.1) by computing the Kendall Tau's coefficient between them. A positive correlation would thus indicate that the more (or less) dominant categories among a group's replied comments are also more (or less) dominant among the other group overall. The correlation coefficients are shown in Table 4. Interestingly, we observe a slight positive correlation between the DRRs for both MHPs and peers with the overall LIWC category usage ranking for MHPs. On the other hand, we see no correlations with the LIWC usage ranks for peers. Intuitively, it appears that for both MHPs and peers, the comments prompting further engagement with the poster appear to reflect the overall dominant linguistic aspects captured by LIWC of MHPs, but not peers. As counseling principles have emphasized the importance of mutual engagement between counselors and clients (Miller and Rollnick, 2012) and other work has shown that higher quality counseling sessions are associated with higher client engagement (Pérez-Rosas et al., 2018), it is compelling to observe associations between linguistic aspects of MHPs with the aspects associated with poster-engagement.

Linguistic Style Matching
Linguistic Style Matching (LSM) measures the extent to which one speaker matches another (Gonzales et al., 2010). It compares two parties' relative use of function words as these words are more indicative of style rather than content (Ireland and Pennebaker, 2010).
Previous studies in counseling conversations have measured LSM to understand the extent that counselors and clients match their language. Pérez-Rosas et al. We follow Nobles et al. (2020)'s approach leveraging Ireland and Pennebaker (2010)'s procedure to measure LSM between support seekers and support providers.
For a text sequence, we compute the percentage of words that belong to each of nine functionword categories c from the LIWC lexicon, which include auxiliary verbs, articles, common adverbs, personal/impersonal pronouns, prepositions, negations, conjunctions, and quantifiers. Then, we compute the LSM of each word category c as shown in Equation 1 where p represents post and r represents the response. The composite LSM score for p and r is the mean of all category LSM scores. For each thread, we separate the MHP and peer replies, LSM c = 1 − abs(cat% p − cat% r ) cat% p + cat% r + .0001 (1) We compute these LSM scores over all data together as well as separately for each subreddit topic (named in Table 3). The resulting scores are shown in Figure 4.
We observe LSM scores vary by topic, and most are similar for peers and MHPs or have overlapping confidence intervals. Compared to their LSMs in other topics, MHPs score lower in SOCIAL, which covers broad social issues that are less specialized to health conditions than the others. However, peers have high LSMs in SOCIAL relative to most other topics, and notably higher LSMs than the MHPs. Additionally, MHPs have higher LSMs than those of peers and relative to their own in communities that cover topics of specific compulsive, mood, and neurodevelopmental disorders (COMPULS., MOOD, and NEURODIV.), communities that orient toward counseling purposes (COPE), or toward advice-seeking communities for health and social concerns (HEALTH). The influences in these results require further investigation, but a possible explanation could be that expert knowledge and experience may offer more benefit to specialized condition-related issues than to broader social issues.

Language Modeling
We further examine differences in word usage by building separate language models for MHPs and peers. We seek to identify language use that is indicative of one group or another by running the language model of one on the data of the other and analyzing words with high perplexity. To run these experiments, we use the language model of Merity et al. (2018a,b), which is a recent LSTMbased language model that achieved state-of-the-art performance by combining several regularization techniques. 8 Our implementation uses a fixed vocabulary of 20,907 tokens for both the peer and MHP language models. This is determined by a minimum count of five across the set of posts from both groups. Each language model is trained for 50 epochs. 9 We use the language model trained on MHP data to find words with high entropy in peer data and vice versa. Since we are concerned with the difference in predictability of words between the MHP and peer language models, we subtract the entropy given by the model trained on that data from the entropy assigned by the model that was not trained on that data. In other words, to find words difficult to predict in B's data, we subtract each word's entropy calculated by the model trained on B from the entropy calculated by the model trained on A as follows, for a set of words, X: (2) If we calculate the entropy difference for each LIWC category and for each assignment of the MHP and peer groups to A and B, we find the highest differences for each category shown in the first and third plots of Figure 5. We find highest entropy scores for words relating to leisure, sex, and numbers when running the MHP language model on peer data. Likewise, when running the peer model on MHP data, the category of discrepancy contains words whose accuracy is improved the least by the peer model, again showing that these words are more indicative of the MHP group.
We perform a similar analysis, creating a language model for posts which have the highest score posts. We measure entropy differences and show the highest scoring categories for each group in the second and fourth plots. Some of the categories indicative of MHP language are also indicative of higher scoring posts; discrepancy, present and future words, and negation words, while other categories like assent and insight words are more dominant in higher scoring posts. The lower scoring posts have the highest entropy differences for some types of words in the peer data, however, we also see that filler, anger, and swear words had the highest entropy differences for the low scoring group. Qualitative example sentences with wordlevel entropy and LIWC annotations are shown in the appendix in Table 7.

Discussion and Future Work
In comparing linguistic aspects of MHP and peer comments, we find MHP tendencies align with established counseling principles and findings in counselor behaviors from recent literature. In particular, they align in the use of words that increase the likelihood of desired perspective shifts associated with clients feeling better after text counseling sessions (RQ2) (Althoff et al., 2016). We also found unique differences in the behavior of MHPs as compared to peers in how they respond to information seekers (RQ1). Although, comments by peers that prompt support-seeker replies also make use of similar word categories to MHPs, which shows that comparing MHPs to peers can offer insight into peer interactions as well.
It is important to note that our analyses rely heavily on the LIWC lexicon. While LIWC and other lexicons can help uncover variational language across groups at an exploratory stage, their use alone does not explain why variations are present.  2020)'s studies have also demonstrated limitations of the use of LIWC when working with word counts to correlate with well-being metrics and an individual's emotional state. We utilize LIWC to understand linguistic behavior differences in conversations with peers and MHPs rather than to evaluate the emotional or mental health state of individuals; however, it is important to consider how these limitations could pertain to our interpretations of their differences, especially as we explore them more deeply in future work. In our study, we explore the patterns we find in the context of previous findings from related literature such as (Althoff et al., 2016) and(Nobles et al., 2020), however it warrants another study into nuanced aspects of the patterns to infer their social functions in support seeking forums in particular.
Although our findings align MHP behaviors with certain counselor behaviors associated with positive outcomes, our analyses do not support claims that MHP behaviors are more beneficial to individuals seeking support; rather, we have shown that the general tendencies of MHPs are in accordance with principles and behaviors demonstrated by counselors in other settings. Understanding the outcomes of these interactions for individual support seekers remains as an area for future work, which could employ surveying methods from prior work to measure perceived empathy in online communities (Nambisan, 2011). Our dataset also enables investigations into whether support-seekers exhibit perspective shifts in interacting with MHPs or peers, and what MHP and peer tendencies are associated with these perspective shifts.
Another direction for future work could focus on modeling social media-specific engagement patterns of MHP and peer interactions. Prior work developed a model that accounts for variables indicating the level of attention threads receive (i.e., thread lengths and number of unique commenters), and variables indicating the degree of interaction between posters and commenters (i.e., time between responses and whether the poster replies to commenters), and used this model to study peerto-peer interactions in online mental health platforms (Sharma et al., 2020); this approach may enable studying supportive interactions in megathreads and threads involving back-and-forth dialog between two or more parties.
More questions arise if we consider MHP tenure and specific domain of expertise (e.g., specializations, licenses, academic degrees). Prior work that studied longitudinal changes in counselor linguistic behaviors indicated that systematic changes occur over time as counselors develop personal styles that are more distinct from other counselors and exhibit more diversity across interactions (Zhang et al., 2019). Future work could model the language longitudinally for MHPs and peers that have longerterm histories of participating in mental health forums to investigate whether systematic changes occur online as well, and if so, whether they reflect similar changes found in prior work.

Limitations and Ethical Considerations
A number of unknowns exist in what we are able to extract from Reddit. For instance, we do not know if users that do not use flairs are mental health professionals. We assume that those who have used the MHP flairs are MHPs and those that have not used them are peers. Additionally, we have grouped all MHP flairs into one group for our analysis, though a more nuanced analysis based on particular professional roles (e.g., psychologists, psychiatrists, social workers) and specializations (e.g., motivational interviewing, cognitive behavioral therapy, family & marriage counseling) may reveal additional trends. Prior work found that disclosing credentials has impacts on engagements that vary by subreddit and linguistic patterns associated with different experience levels and expert domains (Park et al., 2020), thus the effects of disclosing MHP credentials when responding to support-seekers should be investigated.
A classifier or language model used to distinguish between MHPs and peers or to generate the language of either could have negative implications. A generative model that provides feedback to users could generate language that is harmful for those seeking help. Our work could be used to devise a tool to train counselors, however we do not have a direct measure of what type of responses are help-ful or meaningful. In such an application, there is potential to reinforce harmful behaviors due to the inaccuracy of our models. Future studies are needed to determine how to best design a tool to train counselors and how models derived from corpora such as ours correspond to advice that patients find useful.

Conclusion
As the role of social networks is becoming more critical in how people seek health-information, it is important to understand their broader implications to health communication and how health experts can engage to promote the soundest information and offer support to their vulnerable users. By elucidating techniques employed by mental health professionals in their interactions with support-seekers in mental health forums, we have contributed insights toward the broader research direction of understanding how health experts currently engage with these platforms. With evidence that MHP linguistic behaviors associate with further engagement with support-seekers and that these same behaviors are associated with positive counseling conversation outcomes, we have shown that analyzing MHP behavior is a promising direction for better understanding online interaction outcomes, which can further inform forum design and moderation, and expert health promotion efforts.
The code used for our experiments and analyses, and the post ids in our dataset can be accessed at https://github.com/MichiganNLP/ MHP-and-Peers-Reddit.