Citation-Aware Continual Pre-Training for Biomedical Language Models

Masaki Asada; Tomoki Tsujimura; Tatsuya Ishigaki; Shusaku Egami; Ken Fukuda; Hiroya Takamura

Citation-Aware Continual Pre-Training for Biomedical Language Models

Masaki Asada, Tomoki Tsujimura, Tatsuya Ishigaki, Shusaku Egami, Ken Fukuda, Hiroya Takamura

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use <b>...</b> for bold, <i>...</i> for italic, and <url>...</url> for URLs.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

The biomedical literature contains rich structured knowledge, including citation links that encode relationships between scientific studies, but such information is typically ignored in standard language model pre-training. We propose a citation-aware continual pre-training method for decoder-only language models that incorporates citation graph information from PubMed into next-token prediction by placing citation-linked abstract pairs within a shared context. We evaluate our method on multiple biomedical QA benchmarks using two model families. Results show that citation-aware continual pre-training achieves higher average accuracy than both the original base models and citation-unaware pre-training across biomedical tasks.

Anthology ID:: 2026.bionlp-1.32
Volume:: BioNLP 2026
Month:: July
Year:: 2026
Address:: San Diego, California
Editors:: Dina Demner-Fushman, Sophia Ananiadou, Kirk Roberts, Junichi Tsujii
Venues:: BioNLP | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 407–412
Language:
URL:: https://aclanthology.org/2026.bionlp-1.32/
DOI:
Bibkey:
Cite (ACL):: Masaki Asada, Tomoki Tsujimura, Tatsuya Ishigaki, Shusaku Egami, Ken Fukuda, and Hiroya Takamura. 2026. Citation-Aware Continual Pre-Training for Biomedical Language Models. In BioNLP 2026, pages 407–412, San Diego, California. Association for Computational Linguistics.
Cite (Informal):: Citation-Aware Continual Pre-Training for Biomedical Language Models (Asada et al., BioNLP 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.bionlp-1.32.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{asada-etal-2026-citation,
    title = "Citation-Aware Continual Pre-Training for Biomedical Language Models",
    author = "Asada, Masaki  and
      Tsujimura, Tomoki  and
      Ishigaki, Tatsuya  and
      Egami, Shusaku  and
      Fukuda, Ken  and
      Takamura, Hiroya",
    editor = "Demner-Fushman, Dina  and
      Ananiadou, Sophia  and
      Roberts, Kirk  and
      Tsujii, Junichi",
    booktitle = "{B}io{NLP} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.bionlp-1.32/",
    pages = "407--412",
    ISBN = "979-8-89176-434-7",
    abstract = "The biomedical literature contains rich structured knowledge, including citation links that encode relationships between scientific studies, but such information is typically ignored in standard language model pre-training. We propose a citation-aware continual pre-training method for decoder-only language models that incorporates citation graph information from PubMed into next-token prediction by placing citation-linked abstract pairs within a shared context. We evaluate our method on multiple biomedical QA benchmarks using two model families. Results show that citation-aware continual pre-training achieves higher average accuracy than both the original base models and citation-unaware pre-training across biomedical tasks."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="asada-etal-2026-citation">
    <titleInfo>
        <title>Citation-Aware Continual Pre-Training for Biomedical Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Masaki</namePart>
        <namePart type="family">Asada</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tomoki</namePart>
        <namePart type="family">Tsujimura</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tatsuya</namePart>
        <namePart type="family">Ishigaki</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shusaku</namePart>
        <namePart type="family">Egami</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ken</namePart>
        <namePart type="family">Fukuda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hiroya</namePart>
        <namePart type="family">Takamura</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>BioNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dina</namePart>
            <namePart type="family">Demner-Fushman</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sophia</namePart>
            <namePart type="family">Ananiadou</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kirk</namePart>
            <namePart type="family">Roberts</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Junichi</namePart>
            <namePart type="family">Tsujii</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-434-7</identifier>
    </relatedItem>
    <abstract>The biomedical literature contains rich structured knowledge, including citation links that encode relationships between scientific studies, but such information is typically ignored in standard language model pre-training. We propose a citation-aware continual pre-training method for decoder-only language models that incorporates citation graph information from PubMed into next-token prediction by placing citation-linked abstract pairs within a shared context. We evaluate our method on multiple biomedical QA benchmarks using two model families. Results show that citation-aware continual pre-training achieves higher average accuracy than both the original base models and citation-unaware pre-training across biomedical tasks.</abstract>
    <identifier type="citekey">asada-etal-2026-citation</identifier>
    <location>
        <url>https://aclanthology.org/2026.bionlp-1.32/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>407</start>
            <end>412</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Citation-Aware Continual Pre-Training for Biomedical Language Models
%A Asada, Masaki
%A Tsujimura, Tomoki
%A Ishigaki, Tatsuya
%A Egami, Shusaku
%A Fukuda, Ken
%A Takamura, Hiroya
%Y Demner-Fushman, Dina
%Y Ananiadou, Sophia
%Y Roberts, Kirk
%Y Tsujii, Junichi
%S BioNLP 2026
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California
%@ 979-8-89176-434-7
%F asada-etal-2026-citation
%X The biomedical literature contains rich structured knowledge, including citation links that encode relationships between scientific studies, but such information is typically ignored in standard language model pre-training. We propose a citation-aware continual pre-training method for decoder-only language models that incorporates citation graph information from PubMed into next-token prediction by placing citation-linked abstract pairs within a shared context. We evaluate our method on multiple biomedical QA benchmarks using two model families. Results show that citation-aware continual pre-training achieves higher average accuracy than both the original base models and citation-unaware pre-training across biomedical tasks.
%U https://aclanthology.org/2026.bionlp-1.32/
%P 407-412

Download as File

Markdown (Informal)

[Citation-Aware Continual Pre-Training for Biomedical Language Models](https://aclanthology.org/2026.bionlp-1.32/) (Asada et al., BioNLP 2026)

Citation-Aware Continual Pre-Training for Biomedical Language Models (Asada et al., BioNLP 2026)

ACL

Masaki Asada, Tomoki Tsujimura, Tatsuya Ishigaki, Shusaku Egami, Ken Fukuda, and Hiroya Takamura. 2026. Citation-Aware Continual Pre-Training for Biomedical Language Models. In BioNLP 2026, pages 407–412, San Diego, California. Association for Computational Linguistics.