<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="liu-nguyen-2024-rephrasing">
    <titleInfo>
        <title>Rephrasing Electronic Health Records for Pretraining Clinical Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jinghui</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anthony</namePart>
        <namePart type="family">Nguyen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Tim</namePart>
            <namePart type="family">Baldwin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sergio</namePart>
            <namePart type="given">José</namePart>
            <namePart type="family">Rodríguez Méndez</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nicholas</namePart>
            <namePart type="family">Kuo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Canberra, Australia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (\ensuremath&lt;10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.</abstract>
    <identifier type="citekey">liu-nguyen-2024-rephrasing</identifier>
    <location>
        <url>https://aclanthology.org/2024.alta-1.13/</url>
    </location>
    <part>
        <date>2024-12</date>
        <extent unit="page">
            <start>164</start>
            <end>172</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Rephrasing Electronic Health Records for Pretraining Clinical Language Models
%A Liu, Jinghui
%A Nguyen, Anthony
%Y Baldwin, Tim
%Y Rodríguez Méndez, Sergio José
%Y Kuo, Nicholas
%S Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association
%D 2024
%8 December
%I Association for Computational Linguistics
%C Canberra, Australia
%F liu-nguyen-2024-rephrasing
%X Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (\ensuremath<10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.
%U https://aclanthology.org/2024.alta-1.13/
%P 164-172

Markdown (Informal)

[Rephrasing Electronic Health Records for Pretraining Clinical Language Models](https://aclanthology.org/2024.alta-1.13/) (Liu & Nguyen, ALTA 2024)

• Rephrasing Electronic Health Records for Pretraining Clinical Language Models (Liu & Nguyen, ALTA 2024)

ACL

• Jinghui Liu and Anthony Nguyen. 2024. Rephrasing Electronic Health Records for Pretraining Clinical Language Models. In Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association, pages 164–172, Canberra, Australia. Association for Computational Linguistics.