Website owners increasingly restrict AI crawlers through robots.txt. Audits are English-centric and do not stratify results by language. We measure opt-out for 4,900 domains across fourteen languages and three resource tiers in FineWeb and FineWeb2. Among reachable domains, high-tier sites block 28.8 percent, versus 11.2 percent and 9.8 percent in the mid and low tiers; an exact language-partition permutation gives p = 0.008 for the high-low gap. The gradient is concentrated in naming: files naming any of fifteen AI agents fall from 33.6 percent to 10.6 percent across tiers, while named-agent blocking shows no tier ordering. Among files listing conventional crawlers, AI naming falls from 64.0 percent to 19.0 percent, consistent with vocabulary diffusion rather than maintenance failure. Applying current robots.txt retroactively to an earlier crawl, design-weighted document mass behind a Common Crawl opt-out is 31.2 percent for English; tier means are 26.5 percent, 8.9 percent, and 10.7 percent. Reachability falls from 78 percent in English to 39 percent in Uyghur. These results show a gradient in recorded policy that a consent-respecting rebuild would inherit. We release every fetched robots.txt, derived rates, and code.

Pretraining corpus releases report how much duplication their pipelines removed, but we are not aware of one that estimates how much they missed. Measuring the residue is difficult because two recall problems apply at once: detectors miss duplicates they cannot see, and any tractable subsample misses duplicates whose partners were not sampled. In this paper we address both. We 1) run four detectors with distinct failure modes over a corpus and treat them as capture occasions, 2) recover what all four missed through capture-recapture, and 3) invert the sampling rate that produced the subsample, so that the result describes a full 15-trillion-token corpus rather than a convenient slice of it. The design is scored first against a planted population of known size, where it recovers 89.2% of the truth from a 44.5% detected rate while remaining a lower bound. Applied to the 14,868,862-document sample-10BT release of FineWeb, 4.50% of probes have a surviving near-duplicate, and capture-recapture raises that to 4.81% (95% CI 4.37-5.26). Inverting a sampling rate of one document in 1,532 then carries the figure to the corpus and gives a lower bound of 68 on the mean near-duplicate degree of a FineWeb document. Near-duplicate here means lexical overlap at Jaccard 0.35 or above. The bound is on a mean and not on a prevalence, since our design leaves the share of documents with any surviving duplicate anywhere between 7% and 73%. The survivors also carry a positional signature. The two copies of an exact-duplicate pair sit a mean 34.4% of the corpus apart against 33.3% expected under uniform sampling, which supports reading them as residue of FineWeb’s deliberate per-snapshot deduplication. A deduplicated corpus, therefore, is not a duplicate-free corpus, and the size of the gap can now be measured in one streaming pass.

Low-resource subsets of multilingual web corpora often carry large amounts of text in the wrong language, and audits keep rediscovering the problem in each new corpus release. Scoring one corpus at a time cannot say whether a later corpus repeats an earlier one’s errors on the same documents. We join mC4 and FineWeb-2, built independently from overlapping Common Crawl snapshots, at the document level for eight low-resource languages. A URL join and a shift-invariant fingerprint sort the first 250k FineWeb-2 documents per language into three overlap classes: co-selected (same page, near-identical text), page-shared (same page, divergent text), and unmatched. Overlap covers 11% to 74% of a FineWeb-2 subset, so both corpora draw heavily on the same pages, and the shared documents are clean anyway. Under a criterion that flags a document only when two classifiers reject its label, 76.5% of sampled mC4 Sundanese and 58.9% of mC4 Kyrgyz are flagged, against at most 2.7% of co-selected FineWeb-2 documents. We attribute the difference to selection at ingestion. Refiltering mC4 through GlotLID still leaves 0.7-17% of survivors disagreeing with fastText for six languages, and barely moves Sundanese or Yoruba. We release the fingerprints, the flags, and the code: https://github.com/garyzhang1006/wac13-lid-overlap