Sharvin Goyal
Author directory2026
The Crawl Gap: Where Endangered-Language Web Content Is Lost Before It Reaches a Corpus
Sharvin Goyal
Proceedings of the 13th Web-as-Corpus Workshop
Sharvin Goyal
Proceedings of the 13th Web-as-Corpus Workshop
Web-crawled corpora now claim that they have coverage of more than a thousand languages, but their subsets for endangered languages remain very small. Prior audits examine what is inside these corpora. We ask a question that complements this idea: what happens to the endangered-language web content that does exist? We curate and verify an inventory of 40 pages from active institutional websites (tribal language programs, community dictionaries, and learning portals) for 16 endangered North American languages from 7 families, and we attribute losses along the full pipeline: (S1) whether Common Crawl ever crawled the page, (S2) whether language identification (LID) recognizes the captured text, (S3) whether a standard quality-filter cascade can help retain it, and (S4) whether any of it reaches a released corpus (FineWeb-2, GlotCC). Most of the loss happens before corpus construction even begins. These websites present to crawlers almost entirely in English, much of the actual language text is served by client-rendered dictionary applications that expose no text to crawlers, 6 of 16 languages have no label in the standard LID model, and none of the 36 inventory domains contributes a single document to any released per-language subset. We release the inventory, the pipeline, and the per-stage attribution to support both crawl-side and community-side remedies.
Invisible on the Web: An Audit of Low-Resource Muskogean Languages in Web-Crawled Corpora
Sharvin Goyal
Proceedings of the 13th Web-as-Corpus Workshop
Sharvin Goyal
Proceedings of the 13th Web-as-Corpus Workshop
Web-crawled corpora now claim that they retain coverage of 1,000–2,000 languages, but what does such coverage mean for a language family at the extreme low-resource end? We audit the collective presence of the Muskogean family (Choctaw, Chickasaw, Muscogee, Alabama, Koasati, Mikasuki) across seven major web corpora. Only two of the six languages appear in any corpus, and their entire kept content is 28 documents comprising 16 distinct texts, dominated by 19th-century Bible translations; a single document comes from a present-day language community. We exhaustively audit both the kept documents and the 15,108 discarded ones, showing that (i) language identification (LID) assigns Muskogean labels to short non-Muskogean fragments with up to 99.7% persistence under re-identification, so quality filters, not LID, determine corpus content; (ii) those same filters discard about twice as many genuine Choctaw documents as they keep, including contemporary tribal-authored text; and (iii) four Muskogean languages are structurally invisible: no LID label exists, and verified Chickasaw text is silently absorbed into Choctaw. We release our audit annotations and seed-text URL list, and recommend family-level label sets, filter audits on the smallest subsets, and engagement with tribal data sovereignty for web corpus builders.