<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="heddaya-etal-2025-casesumm">
    <titleInfo>
        <title>CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mourad</namePart>
        <namePart type="family">Heddaya</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kyle</namePart>
        <namePart type="family">MacMillan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hongyuan</namePart>
        <namePart type="family">Mei</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chenhao</namePart>
        <namePart type="family">Tan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anup</namePart>
        <namePart type="family">Malani</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: NAACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Luis</namePart>
            <namePart type="family">Chiruzzo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alan</namePart>
            <namePart type="family">Ritter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lu</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Albuquerque, New Mexico</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-195-7</identifier>
    </relatedItem>
    <abstract>This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as “syllabuses.” Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815.We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. We find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains.</abstract>
    <identifier type="citekey">heddaya-etal-2025-casesumm</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-naacl.102</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-naacl.102/</url>
    </location>
    <part>
        <date>2025-04</date>
        <extent unit="page">
            <start>1917</start>
            <end>1942</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions
%A Heddaya, Mourad
%A MacMillan, Kyle
%A Mei, Hongyuan
%A Tan, Chenhao
%A Malani, Anup
%Y Chiruzzo, Luis
%Y Ritter, Alan
%Y Wang, Lu
%S Findings of the Association for Computational Linguistics: NAACL 2025
%D 2025
%8 April
%I Association for Computational Linguistics
%C Albuquerque, New Mexico
%@ 979-8-89176-195-7
%F heddaya-etal-2025-casesumm
%X This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as “syllabuses.” Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815.We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. We find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains.
%R 10.18653/v1/2025.findings-naacl.102
%U https://aclanthology.org/2025.findings-naacl.102/
%U https://doi.org/10.18653/v1/2025.findings-naacl.102
%P 1917-1942

Markdown (Informal)

[CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions](https://aclanthology.org/2025.findings-naacl.102/) (Heddaya et al., Findings 2025)

• CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions (Heddaya et al., Findings 2025)

ACL

• Mourad Heddaya, Kyle MacMillan, Hongyuan Mei, Chenhao Tan, and Anup Malani. 2025. CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1917–1942, Albuquerque, New Mexico. Association for Computational Linguistics.