Communicative-Function-Based Sentence Classification for Construction of an Academic Formulaic Expression Database

Formulaic expressions (FEs), such as ‘in this paper, we propose’ are frequently used in scientific papers. FEs convey a communicative function (CF), i.e. ‘showing the aim of the paper’ in the above-mentioned example. Although CF-labelled FEs are helpful in assisting academic writing, the construction of FE databases requires manual labour for assigning CF labels. In this study, we considered a fully automated construction of a CF-labelled FE database using the top–down approach, in which the CF labels are first assigned to sentences, and then the FEs are extracted. For the CF-label assignment, we created a CF-labelled sentence dataset, on which we trained a SciBERT classifier. We show that the classifier and dataset can be used to construct FE databases of disciplines that are different from the training data. The accuracy of in-disciplinary classification was more than 80%, while cross-disciplinary classification also worked well. We also propose an FE extraction method, which was applied to the CF-labelled sentences. Finally, we constructed and published a new, large CF-labelled FE database. The evaluation of the final CF-labelled FE database showed that approximately 65% of the FEs are correct and useful, which is sufficiently high considering practical use.


Introduction
Formulaic expressions (FEs), such as 'in this paper we propose', are a type of multi-word expressions and are repeatedly used in scientific papers. Some FEs convey a communicative function (CF) of a sentence, which represents intentions of authors. For example, 'in this paper, we propose' conveys the CF of 'showing the aim of the paper'.
Databases comprising CF-labelled FEs are required from a pedagogical perspective (Martinez and Schmitt, 2012), and a computer-based aca- demic writing assistance system 1 that uses such CF-labelled FEs has been proposed (Mizumoto et al., 2017). Several attempts have been made to extract FEs from scientific corpora and categorise them based on CFs (Cortes, 2013;Ädel, 2014;Mizumoto et al., 2017;Morley, n. d.;Simpson-Vlach and Ellis, 2010;Lu et al., 2018). A CFlabelled FE database can be constructed using two main approaches: top-down and bottom-up approaches (Biber et al., 2007;Durrant and Mathews-Aydınlı, 2011). By using the top-down approach, sentences are first assigned CF labels, and then FEs are extracted, while in the case of the bottomup approach, FEs are first extracted and then assigned CF labels. To date, both the approaches have been adopted because CF assignment is performed manually (Table 1). In this paper, we propose a fully automated construction of the CF-labelled FE database, where we consider the top-down approach to be more beneficial (Figure 1). This is because the bottom-up approach requires FEs to be classified, which is difficult because a perfect FE-extraction technique is yet to be realised, and FE embeddings have not been investigated intensively. The top-down approach requires sentence classification, which has highly improved with the recent advancements on pre-trained models.  Table 1: Properties of the existing and proposed methods for the construction of CF-labelled FE databases and the statistics of the databases. The approach of Morley (n. d.) is unknown. For the CF assignment (CF), we adopted supervised machine-learning. The FE extraction (FE) was conducted manually using a corpus-or sentence-level method. Either FEs specific to one discipline were extracted or FEs used in a corpus in which several disciplines were mixed were extracted. The number of documents used for extraction and the extracted FEs of the existing and presented database were shown. Some studies did not disclose the number of documents or FEs. Morley (n. d.) constantly revises the database, and therefore the number of FEs is not fixed.
For CF-based sentence classification, we created a dataset for supervised learning. The dataset consists of a small number of sentences that were assigned CF labels. We collected the sentences from scientific papers of multiple disciplines. By using this dataset, we fine-tuned SciBERT (Beltagy et al., 2019). Additionally, because there are preferences for CF usage depending on disciplines and as the preparation and coverage of all CFs of every discipline are difficult, sentences to which any prepared CF label should not be assigned may appear in a corpus (no-CF sentences). These no-CF sentences will have a negative effect on the classification performance. Based on the recent work on out-of-distribution detection in natural language processing (Hendrycks and Gimpel, 2017;Hendrycks et al., 2020), we used the maximum value of the softmax layer as the threshold to filter no-CF sentences in order to improve the final precision. The experimental results show that the maximum value of the softmax layer works well as the threshold to filter out undesirable sentences.
We carefully considered multidisciplinary problems in the classification. Although the development of a training dataset for every discipline in the world is obviously impossible, demonstrating a successful classification using a single disciplinary dataset is not sufficient for practical use. In this study, we determined whether a model trained on a corpus of one discipline can be applied to that of another discipline. Moreover, the effects of a pre-training dataset were examined by comparing SciBERT and BERT (Devlin et al., 2019). The experimental results show that the classifiers performed fairly well in terms of both in-discipline and cross-discipline data, and the performance was only slightly affected when scientific papers were not used as pre-training data.
For the FE-extraction process, one FE should be extracted from one sentence because CF labels are assigned to each sentence; this is termed as sentence-level approach (see Section 2.2). Therefore, we propose a sentence-level FE extraction method that is based on an existing method (Iwatsuki et al., 2020b). The method consists of three steps: named and scientific entity removal, dependency-structure-based word removal, and word-association-measure-based word removal.
Finally, we created a new, large, multidisciplinary CF-labelled FE database and evaluated it by asking human evaluators whether each instance was assigned a correct CF label and whether an FE was useful for writing a paper. The results show that approximately 65% of the collected FEs are appropriate.
The contributions of our study are as follows: • we created and published the CF-labelled sentence dataset, which is the first dataset for training and evaluation of CF-based classification; • we showed that a simple SciBERT-based neural classifier performed reasonably well for the CF labelling problem; • we showed that the SciBERT classifier can be used even though the discipline of the training data is different from the inferred one; • we proposed an FE extraction method; and • we constructed a CF-labelled FE database with the top-down approach, which is larger than the existing databases but still maintains high quality.
2 Related Work

CFs in Scientific Papers
The CFs of scientific papers were first introduced by Swales (1990), who focused on the CFs in the introduction section. The author proposed a hierarchical structure of CFs, in which move was considered a larger unit of CF and step was a smaller unit belonging to move. He found that the introduction section consists of three moves: 'establishing a territory', 'establishing a niche', and 'occupying the niche'. Each move has several steps, such as 'claiming centrality' and 'presenting research questions or hypotheses' (Swales, 2004). Following his work, a host of studies extended the concept to all parts of a scientific paper. Most studies focused on very limited part of scientific papers; only the introduction (Ozturk, 2007), methods (Lim, 2006;Cotos et al., 2017), results (Basturkmen, 2009;Lim, 2010), discussion sections (Peacock, 2002;Basturkmen, 2012), or the abstracts (Lorés, 2004;Darabad, 2016;Rashidi and Meihami, 2018;Saboori and Hashemi, 2013). The concept was extended to all parts of a scientific paper. For example, Kanoksilapatham (2005) proposed the CF structure of all the sections in biochemistry papers. Cotos et al. (2015) proposed a CF set for all four sections, i.e. introduction, methods, results, and discussion sections. Maswana et al. (2015) compared the usage of the CFs in five engineering fields and found that certain CFs are preferred depending on the discipline.
Argumentative Zoning is a similar concept based on the rhetorical moves (Teufel, 1999). It had seven categories, which were later extended to 15 categories by Teufel et al. (2009) Previous studies on CF-based classification used conditional random fields (Hirohata et al., 2008), a classifier chain with sequential minimum optimisation, Rakel with the J48 algorithm (Dayrell et al., 2012), a Bayes classifier, and a decision tree (Soonklang, 2016). However, these studies only focused on abstracts of scientific papers. Therefore, existing CF-labelled FE lists were created by manually assigning CF labels (Table 1) Bi-LSTM+CRF to classify sentences. However, CF-labelled sentence corpora are yet to be made available to the public.

FE-Extraction Methods
Two approaches are used for extracting FEs: corpus-and sentence-level approaches. Based on the intuition that FEs appear frequently or words composing FE are strongly associated, most studies use the corpus-level approach, in which statistical metrics, such as frequency or mutual information, are applied to a whole corpus. To extract FEs, word n-grams were collected from a whole corpus by using the metrics (Biber et al., 2004;Simpson-Vlach and Ellis, 2010;Kermes, 2012;Mizumoto et al., 2017). However, this approach results in the extraction of an explosive number of overlapping n-grams, thus causing a serious problem in the CFlabelled FE database construction. For instance, suppose 'in this paper we propose', 'this paper we propose a', and 'in this paper we propose a new method' are extracted, a criterion is needed to determine which of these are regarded as FEs; however, determining such a criterion is difficult.
The n-gram lattice method (Brooke et al., 2017) is one approach to address this problem; here, scores of various aspects of formulaicity are first calculated for all word n-grams. Next, an objective function that contains all scores of the n-grams is maximised to determine which n-grams should be disregarded and which should remain. However, this method is still not focused on FEs conveying CFs but on general phrasal expressions; thus, it is thus not suitable for our setting.
The sentence-level approach assumes that one FE occurs in one sentence. Thus, 'in this paper we propose a new method' can be extracted, but 'this paper we propose a' cannot be extracted from a sentence. This approach is also useful for extracting FEs with a slot (Vincent, 2013), into which some words can be inserted, such as 'however, * have not been reported'. This setting is regarded as a sequence-labelling problem, in which each word of a sentence is labelled as either formulaic or non-formulaic. Liu et al. (2016) proposed removing topic-specific words as non-formulaic words, using latent Dirichlet allocation. They used a corpus consisting of papers from various disciplines, and tried to remove discipline-specific vocabulary. Thus, this is not suitable for extracting disciplinespecific FEs. Iwatsuki et al. (2020b) proposed re-moving scientific and named entities in addition to dependency-based word removal.
The evaluation of the FE extraction model is another problem. Brooke et al. (2015) pointed out that the comparison of newly extracted FEs with existing reference is unreasonable because if a reference is on point, a new lexicon need not be created. Thus, Iwatsuki et al. (2020b) proposed evaluating FE extraction methods by a CF-based sentence retrieval task as an extrinsic task based on the idea that FEs convey a CF of a sentence. Table 1 describes the existing CF-labelled FE databases. Previous studies have shown that FEs are discipline-specific, and the resource of academic vocabulary should be presented for each discipline (Hyland and Tse, 2007;Liu, 2012). Thus, the development of CF-labelled FE databases for each discipline is important; however, many studies have focused on general FEs, which were extracted from a mixed corpus consisting of scientific papers on multiple disciplines. Some studies adopted the discipline-specific approach; Mizumoto et al. Hence, we contend that the automated CF-based classification is helpful for constructing a large, comprehensive CF-labelled FE database. In this study, we developed a discipline-specific database based on large corpora of scientific papers from four disciplines.

Corpora of Scientific Papers
In this study, we considered the corpora which satisfy the following conditions. First, because we use full text of scientific papers and have made all the data public, papers must be open access. Second, to construct a comprehensive database, the corpora size is important. Third, for cross-discipline analyses, a discipline-specific journal is preferred to a multidisciplinary journal. We selected a corpus containing at least 10,000 papers.
Under these three conditions and based on the diversity of the disciplines, we selected four corpora: ACL Anthology Sentence Corpus 2 for computational linguistics (CL), Molecules 3 for chemistry (Chem), Oncotarget 4 for oncology (Onc), and Frontiers in Psychology 5 for psychology (Psy). Each corpus comprises more than 10,000 papers and is open access to full text (creative commons licence).
For pre-processing, we performed sentence splitting using ScispaCy (Neumann et al., 2019) and replaced citations and mathematical formulae with a special token. By using a simple rule-based method, section labels were normalised into five classes: introduction, methods, results, discussion, and other. Each sentence was assigned a section label; we did not use sentences belonging to the 'other' class. The numbers of sentences and documents are listed in Table 2

CF Set and CoreFEs
Till date, there is no established CF set, and some CFs are not used or are frequently used in a specific discipline. Proposing a new CF set is beyond the scope of this study; however, we must select a CF set. We adopted the CF set proposed by Iwatsuki et al. (2020a), which was based on CFs used in Academic Phrasebank (Morley, n. d.). Table 3 describes the numbers of CFs in each section. (All the CFs are listed in Table 13 in the appendix.) CoreFE is an FE that is shortened so that it can be used as a query for sentence retrieval (Generally, longer phrases result in few or no results in sentence retrieval). We used CoreFEs to create the CF-labelled sentence dataset.

CF-Labelled Sentence Dataset
For the CF-based classification, we created a sentence dataset by using the aforementioned corpora.
To effectively collect labelled sentences, we used the following procedures. First, the CoreFEs were Section #CFs Introduction 11 Methods 6 Results 6 Discussion 9 used as queries to retrieve sentences from the corpora. Although the CoreFEs have CF labels, the retrieved sentences may not always have the same CFs. Next, we used Amazon Mechanical Turk (AMT) to check if each sentence was assigned correct labels; this process was three-fold. First, a correct set of sentences was prepared. Two experts were asked whether the sentences in the correct set were correctly labelled, and the sentences whose labels were judged incorrect by at least one expert were removed. Another set of sentences, called the incorrect set, was prepared, in which the same sentences were randomly assigned incorrect labels. Second, by using these sets, a pilot test was conducted on AMT. Five annotators were recruited and asked to check whether the labels were correct or not. Based on this pilot test, we determined the threshold to cut off sentences. Finally, a larger set of sentences was prepared, which was different from the set used in the pilot test. Another five annotators were asked to perform the same task on the set. The final dataset comprises the sentences satisfying the threshold.

Classifier
We assigned each sentence a CF label, and this task can be regarded as a CF-based sentenceclassification problem. In addition, we used SciB-ERT (Beltagy et al., 2019) with an additional linear layer for classification. We split the CFlabelled sentence dataset into training/development and evaluation sets so that four sentences for each CF were in the evaluation set. Then, we conducted five-fold cross validation using the training/development set for parameter tuning. Subsequently, we fine-tuned the classifier and evaluated the classification accuracy.
Because CF sets in scientific papers have not been established, the CF set we used cannot satisfactorily cover all sentences written in papers. Additionally, pre-processing errors, such as sentence splitting, sometimes result in no-CF sen-tences. Thus, in some scenarios, no CF should be assigned to a sentence and no-CF sentences must be removed. The no-CF class is not contained in the training dataset; this problem is regarded as the out-of-distribution detection problem. Although the maximum value of the softmax layer is not a perfect metrics for out-of-distribution detection, pre-trained transformers, such as BERT and RoBERTa, with a softmax layer are good detectors of out-of-distribution data (Hendrycks and Gimpel, 2017;Hendrycks et al., 2020).
To manage the no-CF sentences, we used the maximum softmax value of the classifier, and verified its performance. The verification was performed in the same manner as the creation of the CF-labelled sentence dataset. That is, we asked five AMT annotators whether the output label was correct. The threshold was also the same: 5/5.

Multidisciplinary Perspectives
To create a multidisciplinary database, the classification must be applied to various disciplinary texts. As it is costly to create a training dataset manually for each discipline, we tested whether the classifier trained on a dataset of one discipline can be immediately applied to the datasets of other disciplines.
SciBERT was trained on scientific papers from Semantic Scholar 6 (Beltagy et al., 2019). The corpora used in this study are open access and were also included in Semantic Scholar. Thus, we hypothesise that the cross-disciplinary adaptation is successful because the sentences are (partly) contained in the pre-training dataset. Therefore, the method cannot be applied to disciplines that are not covered by the pre-training dataset. To verify this hypothesis, we compared SciBERT to BERT, which was pre-trained on the book corpus and Wikipedia and not on scientific papers (Devlin et al., 2019), for cross-discipline sentence classification.

FE Extraction
To extract FEs, we propose a method based on Iwatsuki et al. (2020b), which is a sentence-level method; one FE was extracted from one sentence. We applied this method, which comprises three steps, to the classified sentences.
In the first step, the named and scientific entities are removed from a sentence. The entity recognition was performed using SpERT (Eberts and Ulges, 2020), which sits atop the leader-board of NER tasks for scientific entities 7 . For training, we used CoNLL04 (Roth and Yih, 2004), a corpus labelled with general-purpose named entities, and SciERC (Luan et al., 2018), a corpus of scientific papers labelled with scientific entities. The CoNLL04 labels are location, organisation, people, and other; SciERC labels are task, method, evaluation metric, material, other scientific terms, and generic. By removing the named entities, a sentence was split into several spans.
In the second step, we used the dependency structure of a sentence analysed by Stanford CoreNLP (Qi et al., 2018). Words that were neither in the span containing a sentence's root nor organised by the root were then removed. The assumption here was that FEs representing CFs of sentences appeared in the structural centres in the sentence dependency structures (Iwatsuki et al., 2020b).
Steps 1 and 2 work well if several named entities are contained in a sentence; otherwise, an almost full sentence is produced, which is too long to be an FE. Thus, we propose an additional filtering step that further removes non-relevant generic terms from the candidate FE spans. This is based on the assumption that each word of an FE is strongly associated with each other. Thus, the association between fragments of an FE should be strong. For instance, 'in this paper we' and 'propose' are strongly associated, while 'in this paper we' and 'talk' are not.
On the basis of this observation, we first extracted all pairs of an n-gram and its neighbour word from each candidate span obtained after Step 2. For example, pairs such as ('in this', 'paper') or ('paper we', 'propose') are obtained when n = 2. Next, for each pair, we calculated the association measures between an n-gram and a neighbour word. We used the local mutual information (LMI), which is formalised as follows: where a and b denote a word, a, b denotes the cooccurrence of the words, p(a) is a probability of occurrence of a, and f (a) is a frequency of a in a corpus (Evert, 2005). Finally, the pairs with the top k scores were labelled as an FE. To avoid generating FEs that are too short, this third process was 7 https://paperswithcode.com/sota/ named-entity-recognition-ner-on-scierc CF: Suggestion of future work Sentence: In the future, we plan to explore how to combine more features such as part-of-speech tags into our model. applied only when the length of the resulting word sequence of Step 2 was more than k words. From our preliminary experiments, we determined to use (n, k) = (2, 7).
Because FEs are assumed to be used as they are, we did not lemmatise them. Formulaicity sometimes does not allow the replacement of a word in an FE with another word or flection. For example, tenses can be section-specific (present or past): 'in this paper we proposed' rarely occurs in the introduction sections. Formulaicity also avoids grammatical errors such as 'little researches have been done'. Many previous studies did not lemmatise FEs (Simpson-Vlach and Ellis, 2010;Mizumoto et al., 2017;Pan et al., 2016;Esfandiari and Barbary, 2017).

Constructing CF-Labelled FE Database
We created the CF-labelled FE database using the following steps.
Step 1: CF labels were assigned to each sentence in a corpus and no-CF sentences were removed.
Step 2: FEs were extracted from each sentence.
Step 3: Noisy FEs were filtered out. If an FE was assigned multiple CF labels, only one CF was selected by majority voting. If none of the CFs took the majority, the FE was removed. Any CF-labelled FE occurring less than three times was also removed.
We evaluated the final database from two perspectives: whether a sentence was assigned a correct label and whether an FE was useful for writing a scientific paper.
The evaluation was conducted on the AMT. A sentence and its CF label were shown to evaluators, and an FE was highlighted in the sentence (see Figure 2). The evaluators were asked whether the sentence conveyed the CF and whether the FE was useful. Each FE was annotated by five evaluators, and if it was not evaluated by all as correct or useful, it was regarded as incorrect or useless.

CF-Labelled Sentence Dataset
The correct and incorrect sets consist of 55 sentences. The results of the pilot test are shown in Table 4. Accordingly, we set the threshold to 5/5 because high precision was important for creating the FE database rather than recall, and the strictest threshold did not significantly reduce the sentences. Table 5 lists the total number of sentences.

CF-Based Sentence Classification
The classification results are shown in Table 6. SciBERT worked well, which implies that this BERT-based classifier has the ability to capture CFs of sentences. We also verified with SciBERT whether the maximum value of the softmax layer can be used as the threshold to filter out no-CF sentences. We first classified all the sentences in the corpora, and then split the classified sentences into six categories based on the maximum softmax score: [0.00, 0.60], (0.60, 0.70], (0.70, 0.80], (0.80, 0.90], (0.90, 0.99], (0.99, 1.00]. Next, we randomly sampled 100 sentences from each range, and the sentences were evaluated by five annotators on AMT. The evaluation method was the same as that used for collecting the CF-labelled sentences. The accuracy of each range is shown in Table 7. For database construction, we removed the sentences with a score of 0.80 or lower.   Table 7: Accuracy scores of each range of the maximum value of the softmax layer, and the proportion of sentences in the corpora.

Multidisciplinary Perspective
We tested whether SciBERT trained on one discipline can be applied to different disciplines. The results are shown in Table 8. We also tested the effects of the pre-trained dataset by comparing the results of SciBERT and BERT. Table 9 and 10 show the BERT results; compared with the results shown in Table 6 and 8, the two models did not show a considerable difference.

Constructing CF-Labelled FE Database
The CF-labelled FE database was evaluated by sampling 200 FEs. The results are shown in Table 11.
Incorrect sentence-CF pairs were obtained because the classifier made errors and some sentences were not a complete sentence. An example of an incomplete sentence is 'of three independent experiments.'; this was produced because of the error of sentence splitting.    'plays a crucial role in' (CF: Showing the importance of the topic) and 'no significant differences were detected in' (CF: Description of the results), while 'et al demonstrated that' (CF: Showing background provided by past work) and 'is to use a' (CF: Showing brief introduction to the methodology) were judged useless.
The statistics of the database are shown in Table 2. To show discipline-specific FEs, we calculated odds ratio for each CF of each discipline. Table 12 illustrates the top 5 high odds ratio FEs in the 'description of the process' CF in the introduction section. These FEs are not considered rare, as some of them occur more than a thousand times in a corpus. The differences between disciplines are relative, and these results may change if another corpus of a different discipline is added; however, preference for FEs still exists across disciplines. This reinforces the previous claim that FEs are discipline-specific (Hyland and Tse, 2007;Hyland, 2008;Durrant, 2015;Jalilifar et al., 2016). All the discipline-specific FEs are listed in Table 15 in the appendix.   Table 12: Examples of discipline-specific FEs. The complete list is provided in the appendix. All the FEs are lower-cased. The number of occurrences of each FE in the corpus is also shown.

CF-Based Sentence Classification
The classification accuracy was quite high, and thus the results can be a good baseline for a CFbased sentence classification task. We published the dataset so that other researchers can tackle the classification task.
The no-CF detection worked fairly. From Table 7 it can be said that the maximum value is often too high; 30% of the CF labels assigned scores higher than 0.99 were incorrect. However, much lower (≤ 0.80) scores tended to cause lower accuracy. Thus, this approach is useful to improve overall precision, which is more important to construct a FE-CF database than recall.

Problems in Multidisciplinary Data
We raised two questions: Can the classifier trained on one discipline be applied to other disciplines? Does the pre-training data affect the classification performance?
The results of the sentence classification imply that the SciBERT classifier trained on a dataset of one discipline can be applied to datasets of other disciplines. This mitigates the labour of creating a training dataset for all other disciplines. Therefore, we argue that to create another FE-CF database of another discipline, the CF-labelled sentence dataset we created can be used as a training dataset.
The comparison of SciBERT (Table 6) and BERT (Table 9) denied our hypothesis that the cross-discipline adaptation worked as long as the discipline was included in pre-training data. Therefore, the ability of discipline adaptation does not come from the pre-training dataset, which implies that the classifier could be used irrespective of whether a discipline is covered by the pre-training dataset.

Quality of the FE-CF Database
The results of the evaluation ( Consider another case in which users use an FE as a query to obtain some example sentences that play the role of a specific CF. In this case, the evaluation results imply that approximately 90% (130/142) of the retrieved sentences are satisfying results. In some cases, the same FEs appear in different CF categories. For example, 'play critical roles in' is used in 'Showing the importance of the topic (introduction)' and 'Showing background provided by past work (discussion)'. Thus, compared to the mere collection of FEs, the addition of CF labels to FEs is proved to be more helpful.

Conclusion
In this paper, we proposed the fully-automated construction of a CF-labelled FE database, by solving the problem of CF-based sentence classification. We carefully considered a practical case of creating a FE database of other disciplines. The experimental results showed that the proposed classifi-cation method and dataset can be utilised to construct FE databases for disciplines different from those that we used. We proposed the FE extraction method that utilised the named and scientific entity removal, dependency-structure-based word removal, and word-association-measure-based word removal. Combining the proposed methods, we finally constructed the new CF-labelled FE database. The CF-labelled sentence database and the CFlabelled FE database are available on our website 8 . We expect that the proposed database could be used by pedagogical practitioners and for computeraided academic-writing assistance such as sentence retrieval and automated proofreading.

A Dataset and Databases
On our website 9 , we published the following dataset and databases: 1. The CF-labelled sentence dataset for training and evaluation, 2. The CF-labelled sentence database, which was constructed by applying SciBERT classifier to every sentence in the corpora we used, and 3. The CF-labelled FE database, which was constructed by applying the proposed FE extraction method to the CF-labelled sentence database.
These data were formatted in tab-separated text. In the CF-labelled sentence dataset, a line consists of an ID and a sentence. In the CF-labelled sentence database, a line consists of a sentence ID (from the corpora), an ID, the maximum softmax value, and a sentence. In the CF-labelled FE database, a line consists of a CF, an FE, and the number of appearance in the corpus.

C General and Discipline-Specific FEs
General FEs are FEs that appear commonly in multiple disciplines. We calculated the average rank of each FE and Table 14 lists the top-5 general FEs for each CF. For most of the CFs, general FEs were not found. We also calculated the odds ratio and Table 15 lists the top-5 discipline-specific FEs for each CF. Some CFs did not happen in a corpus.  to the best of our knowledge no Showing the importance of the research to the best of our knowledge this is the first Showing brief introduction to the methodology is based on Showing brief introduction to the methodology is based on the Showing brief introduction to the methodology are based on Showing brief introduction to the methodology is to use Showing the outline of the paper the paper is organized as follows CF FE Showing the outline of the paper this paper is organized as follows Showing the outline of the paper the paper is structured as follows Showing the outline of the paper this paper is structured as follows Showing the outline of the paper the rest of the paper is structured as follows Section: Results Description of the results there was no significant difference in Description of the results we found that the Description of the results there was no significant difference between the Description of the results there was no significant difference between Description of the results we found that Restatement of the aim or method we used the Restatement of the aim or method we compared the Restatement of the aim or method we used a Restatement of the aim or method we performed a Restatement of the aim or method was performed using Reference to tables or figures as shown in Describing interesting or surprising results it is interesting to note that Describing interesting or surprising results it is interesting to note that the Summary of the results these results suggest that Summary of the results this suggests that Summary of the results this suggests that the Summary of the results this indicates that the Section: Discussion Restatement of the results we found that the Restatement of the results we found that Restatement of the results it is interesting to note that Restatement of the results it is worth noting that the Restatement of the results it is important to note that the Suggestion of hypothesis our results suggest that Explanation for findings can be explained by the fact that Explanation for findings this is due to the fact that Unexpected outcome it is not surprising that the Implications of the findings this raises the possibility that Chem Onc Psy of this paper is organized as follows the rest of the paper is organized as follows the rest of * paper is organized as follows the remainder of the paper is organized as follows remainder of this paper is structured as follows the rest of this paper is organized as follows paper is organized as follows section 2 can be divided into the contributions of this paper are as follows remainder of this paper is organized as follows there was a significant interaction between outperforms all other was assigned to the showed * figure cite-there was a main effect of CF: Describing interesting or surprising results this is due to the fact that it is worth noting that the interestingly we found that et al citeis due to the fact that the it is important to mention that interestingly we observed that it should be noted that the it should be noted that the it is worth mentioning that the indeed we found that and * citewe call this best of our knowledge this is the first report interestingly we found that the however it is important to note that this can be explained by the fact that it is important to note that moreover * figure cite-et al cite-found CF: Summary of the results this shows that our the results indicated that taken together these data demonstrate that this indicates that * likely this result shows that the proposed these results are in agreement with those taken together these results demonstrated that this pattern is consistent with the from these results we can conclude that these results are in accordance with taken together these findings indicate that therefore hypothesis 3 is supported this suggests that for this indicated that the taken together our data suggest that this suggests that during both meditation conditions saline these results demonstrate that the proposed the results show that the these results suggest that * promotes this suggests that the * had