@inproceedings{kulkarni-etal-2021-lmsoc-approach,
    title = "{LMSOC}: An Approach for Socially Sensitive Pretraining",
    author = "Kulkarni, Vivek  and
      Mishra, Shubhanshu  and
      Haghighi, Aria",
    editor = "Moens, Marie-Francine  and
      Huang, Xuanjing  and
      Specia, Lucia  and
      Yih, Scott Wen-tau",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.findings-emnlp.254",
    doi = "10.18653/v1/2021.findings-emnlp.254",
    pages = "2967--2975",
    abstract = "While large-scale pretrained language models have been shown to learn effective linguistic representations for many NLP tasks, there remain many real-world contextual aspects of language that current approaches do not capture. For instance, consider a cloze test {``}I enjoyed the {\_}{\_}{\_}{\_}{\_} game this weekend{''}: the correct answer depends heavily on where the speaker is from, when the utterance occurred, and the speaker{'}s broader social milieu and preferences. Although language depends heavily on the geographical, temporal, and other social contexts of the speaker, these elements have not been incorporated into modern transformer-based language models. We propose a simple but effective approach to incorporate speaker social context into the learned representations of large-scale language models. Our method first learns dense representations of social contexts using graph representation learning algorithms and then primes language model pretraining with these social context representations. We evaluate our approach on geographically-sensitive language modeling tasks and show a substantial improvement (more than 100{\%} relative lift on MRR) compared to baselines.",
}