@inproceedings{zhang-2026-many,
title = "How Many Duplicates Are Left? Capture-Recapture and Sampling Inversion for Residual Duplication in {F}ine{W}eb",
author = "Zhang, Gary",
editor = "Ortiz Suarez, Pedro and
Burchell, Laurie and
Ljube{\v{s}}i{\'c}, Nikola and
Scherrer, Yves and
Laippala, Veronika and
Dini{\'c}, Vuk and
Vaughan, Thom",
booktitle = "Proceedings of the 13th Web-as-Corpus Workshop",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.wac-1.16/",
pages = "170--181",
ISBN = "979-8-89176-481-1",
abstract = "Pretraining corpus releases report how much duplication their pipelines removed, but we are not aware of one that estimates how much they missed. Measuring the residue is difficult because two recall problems apply at once: detectors miss duplicates they cannot see, and any tractable subsample misses duplicates whose partners were not sampled. In this paper we address both. We 1) run four detectors with distinct failure modes over a corpus and treat them as capture occasions, 2) recover what all four missed through capture-recapture, and 3) invert the sampling rate that produced the subsample, so that the result describes a full 15-trillion-token corpus rather than a convenient slice of it. The design is scored first against a planted population of known size, where it recovers 89.2{\%} of the truth from a 44.5{\%} detected rate while remaining a lower bound. Applied to the 14,868,862-document sample-10BT release of FineWeb, 4.50{\%} of probes have a surviving near-duplicate, and capture-recapture raises that to 4.81{\%} (95{\%} CI 4.37-5.26). Inverting a sampling rate of one document in 1,532 then carries the figure to the corpus and gives a lower bound of 68 on the mean near-duplicate degree of a FineWeb document. Near-duplicate here means lexical overlap at Jaccard 0.35 or above. The bound is on a mean and not on a prevalence, since our design leaves the share of documents with any surviving duplicate anywhere between 7{\%} and 73{\%}. The survivors also carry a positional signature. The two copies of an exact-duplicate pair sit a mean 34.4{\%} of the corpus apart against 33.3{\%} expected under uniform sampling, which supports reading them as residue of FineWeb{'}s deliberate per-snapshot deduplication. A deduplicated corpus, therefore, is not a duplicate-free corpus, and the size of the gap can now be measured in one streaming pass."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-2026-many">
<titleInfo>
<title>How Many Duplicates Are Left? Capture-Recapture and Sampling Inversion for Residual Duplication in FineWeb</title>
</titleInfo>
<name type="personal">
<namePart type="given">Gary</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 13th Web-as-Corpus Workshop</title>
</titleInfo>
<name type="personal">
<namePart type="given">Pedro</namePart>
<namePart type="family">Ortiz Suarez</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Laurie</namePart>
<namePart type="family">Burchell</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Nikola</namePart>
<namePart type="family">Ljubešić</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yves</namePart>
<namePart type="family">Scherrer</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Veronika</namePart>
<namePart type="family">Laippala</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Vuk</namePart>
<namePart type="family">Dinić</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Thom</namePart>
<namePart type="family">Vaughan</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-481-1</identifier>
</relatedItem>
<abstract>Pretraining corpus releases report how much duplication their pipelines removed, but we are not aware of one that estimates how much they missed. Measuring the residue is difficult because two recall problems apply at once: detectors miss duplicates they cannot see, and any tractable subsample misses duplicates whose partners were not sampled. In this paper we address both. We 1) run four detectors with distinct failure modes over a corpus and treat them as capture occasions, 2) recover what all four missed through capture-recapture, and 3) invert the sampling rate that produced the subsample, so that the result describes a full 15-trillion-token corpus rather than a convenient slice of it. The design is scored first against a planted population of known size, where it recovers 89.2% of the truth from a 44.5% detected rate while remaining a lower bound. Applied to the 14,868,862-document sample-10BT release of FineWeb, 4.50% of probes have a surviving near-duplicate, and capture-recapture raises that to 4.81% (95% CI 4.37-5.26). Inverting a sampling rate of one document in 1,532 then carries the figure to the corpus and gives a lower bound of 68 on the mean near-duplicate degree of a FineWeb document. Near-duplicate here means lexical overlap at Jaccard 0.35 or above. The bound is on a mean and not on a prevalence, since our design leaves the share of documents with any surviving duplicate anywhere between 7% and 73%. The survivors also carry a positional signature. The two copies of an exact-duplicate pair sit a mean 34.4% of the corpus apart against 33.3% expected under uniform sampling, which supports reading them as residue of FineWeb’s deliberate per-snapshot deduplication. A deduplicated corpus, therefore, is not a duplicate-free corpus, and the size of the gap can now be measured in one streaming pass.</abstract>
<identifier type="citekey">zhang-2026-many</identifier>
<location>
<url>https://aclanthology.org/2026.wac-1.16/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>170</start>
<end>181</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T How Many Duplicates Are Left? Capture-Recapture and Sampling Inversion for Residual Duplication in FineWeb
%A Zhang, Gary
%Y Ortiz Suarez, Pedro
%Y Burchell, Laurie
%Y Ljubešić, Nikola
%Y Scherrer, Yves
%Y Laippala, Veronika
%Y Dinić, Vuk
%Y Vaughan, Thom
%S Proceedings of the 13th Web-as-Corpus Workshop
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-481-1
%F zhang-2026-many
%X Pretraining corpus releases report how much duplication their pipelines removed, but we are not aware of one that estimates how much they missed. Measuring the residue is difficult because two recall problems apply at once: detectors miss duplicates they cannot see, and any tractable subsample misses duplicates whose partners were not sampled. In this paper we address both. We 1) run four detectors with distinct failure modes over a corpus and treat them as capture occasions, 2) recover what all four missed through capture-recapture, and 3) invert the sampling rate that produced the subsample, so that the result describes a full 15-trillion-token corpus rather than a convenient slice of it. The design is scored first against a planted population of known size, where it recovers 89.2% of the truth from a 44.5% detected rate while remaining a lower bound. Applied to the 14,868,862-document sample-10BT release of FineWeb, 4.50% of probes have a surviving near-duplicate, and capture-recapture raises that to 4.81% (95% CI 4.37-5.26). Inverting a sampling rate of one document in 1,532 then carries the figure to the corpus and gives a lower bound of 68 on the mean near-duplicate degree of a FineWeb document. Near-duplicate here means lexical overlap at Jaccard 0.35 or above. The bound is on a mean and not on a prevalence, since our design leaves the share of documents with any surviving duplicate anywhere between 7% and 73%. The survivors also carry a positional signature. The two copies of an exact-duplicate pair sit a mean 34.4% of the corpus apart against 33.3% expected under uniform sampling, which supports reading them as residue of FineWeb’s deliberate per-snapshot deduplication. A deduplicated corpus, therefore, is not a duplicate-free corpus, and the size of the gap can now be measured in one streaming pass.
%U https://aclanthology.org/2026.wac-1.16/
%P 170-181
Markdown (Informal)
[How Many Duplicates Are Left? Capture-Recapture and Sampling Inversion for Residual Duplication in FineWeb](https://aclanthology.org/2026.wac-1.16/) (Zhang, WAC 2026)
ACL