We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input.

We introduce the CreateDebate Arguments Dataset, a large-scale, unfiltered collection of 383,220 arguments from 7,435 of the most active for/against debates on the CreateDebate online debate platform, spanning 16 years of activity. Unlike existing computational argumentation resources, the dataset combines large scale with hierarchical debate-tree structure, author-assigned stance labels, community vote scores, and relational tags that explicitly mark when one argument disputes another. Because the full dataset is unfiltered, we release it under a gated access agreement and additionally release an openly available, manually curated subset of 84,872 arguments from 578 debates. To demonstrate the dataset’s utility, we define two counter-argument retrieval benchmarks on the curated subset: a targeted setting that uses the dispute tag as author-verified ground truth, and a general setting in which any opposing-side argument is a valid counter. Evaluating a BM25 baseline, a frozen bi-encoder, and a fine-tuned bi-encoder, we find consistent but bounded gains from fine-tuning on disputed pairs, a negative effect of debate-level context on within-debate retrieval, and limited cross-task transfer between the targeted and general settings. A qualitative analysis surfaces failure modes specific to naturally occurring debate. The dataset and models are released to the community.

News outlets write in recognizable ways, but how much of that style survives in a single headline once the topic is fixed, and what is it made of? We study outlet attribution from French news headlines under increasingly strict topic control, from category balancing to matched-event comparison. On an ACPM-anchored corpus of 170,160 headlines from ten national outlets across three hard-news categories, with outlets selected through external circulation/audience rankings, a fine-tuned CamemBERT identifies the publisher of an entity-masked headline in a future, unseen year at nearly six times chance; masking outlet self-references barely changes this. We then decompose the fingerprint within the interpretable character model (which alone is surface-strippable). Staged removal of surface form and named entities attributes it, in order-dependent shares, largely to presentation (casing, punctuation, typographic house style) and to a lexical residual that survives removing both, with a smaller share carried by which entities are named. An interpretable tone family adds almost nothing at the single-headline level, so the residual is not reducible to tone here. Category control alone cannot separate editorial form from an outlet’s story selection, so as a final check we hold the reported event constant, clustering same-event headlines across outlets and comparing them within cluster with a human-validated fixed-effects estimator. Both pre-specified attribution contrasts fall to near zero, while one validated difference survives: BFMTV foregrounds institutional and corporate actors as grammatical agents more than its peers. The result shows how much control is required before outlet differences can be read as framing rather than agenda. We release identifiers, hashes, code, derived features, the framing codebook, and gold annotations.

The world’s 7,000+ languages vary widely in the availability of resources for natural language processing, motivating efforts to systematically categorize them by their degree of resourcefulness. A similar disparity exists among programming languages; however, no resource-tier taxonomy has been established for code. As large language models grow increasingly capable of generating code, such a taxonomy becomes essential. To fill this gap, we present the first reproducible programming-language resource classification, grouping 646 languages into four tiers. We show that only 1.9% of languages (Tier 3, High) account for 74.6% of all tokens in seven major web-derived corpora, while 71.7% of languages (Tier 0, Scarce) contribute just 1.0%. Statistical analyses of within-tier inequality, dispersion, and distributional skew confirm that this imbalance is both extreme and systematic. Our results provide a principled framework for dataset curation and tier-aware evaluation of multilingual large language models, as well as the first quantitative picture of code availability on the web.

This work investigates content persistence across two independently constructed versions of the CLASSLA-web corpora for seven South Slavic languages, crawled approximately two years apart, with available document-level genre and topic metadata. Persistence is defined as the presence of a near-duplicate document in both versions of the corpora, which occurs on average in 19% of cases. Genre and news-topic persistence rankings are moderately consistent across languages. Web domain availability after two years is moderately associated with genre persistence (r=.49, R²=.24), but not with topic persistence within news (r=-.04). Topic-level differences therefore appear to arise mainly within surviving news websites, where reference-oriented content tends to persist more than event-bound content.

Large-scale web corpora are valuable resources for natural language processing (NLP), linguistic research, and language model training, particularly for digitally low-resource languages such as Persian. However, web-derived text often contains document-level noise, even after quality-filtering pipelines. We present HPLT3 Persian Refined (HPLT3-PerRef), a refined Persian web corpus containing 105.9 billion tokens and 102.37 million documents, obtained by applying a supervised quality-filtering pipeline to the Persian subset of HPLT3. We manually annotated 6,144 Persian web documents and trained a multilingual transformer-based classifier to distinguish high- and low-quality documents. The classifier removed approximately 20% of the original corpus while preserving broad domain coverage and complements HPLT3’s heuristic-based Web Document Score (WDS) by identifying additional low-quality documents. We publicly release the corpus, the trained classifier, and the complete source code.

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl’s per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl’s observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.

AI-generated overviews are becoming an increasingly prominent layer of search interfaces, yet their behavior in Chinese-language search remains underexplored. We conduct a cross-language audit of AI overview behavior on Baidu and Google using English queries sampled from MS MARCO and their translated Chinese counterparts. Our analysis examines when overviews are triggered across platform-language settings, which host domains receive visible exposure in Chinese overviews, how concentrated that exposure is, and how source overlap varies across settings. We also compare the embedding-based semantic similarity of generated answers for matched query intents. The results reveal substantial differences across platform-language settings in overview availability and visible source exposure. At the aggregate level, the settings exhibit low overlap in visible host-domain inventories, while matched-query answers remain broadly semantically similar. These findings indicate that answer-level semantic similarity and aggregate source exposure capture distinct dimensions of AI-mediated search. Evaluations of AI search should therefore consider not only the content of generated answers but also how source visibility is distributed across platforms, languages, and information environments.

Machine translation of English web text is a default recipe for building multilingual and low-resource corpora. We show that this recipe systematically damages the structured spans embedded in web text—URLs, email addresses, numbers and identifiers, code, markup and quotation delimiters—and that fluency metrics do not reveal the damage. Translating 56,163 span-bearing segments from a CC-MAIN-2026-30 Common Crawl sample into nine typologically diverse languages with NLLB-200, we find that 12.6% of 73,881 structured spans do not survive, ranging from 91.1% for numbers to 50.6% for quotation delimiters. Damage is strongly language-dependent (18.7 points), but in the opposite direction to the obvious prediction: the highest-resource targets are damaged most, an inversion that survives controlling for the systems’ tendency to copy low-resource input through untranslated. Round-trip fluency barely tracks any of this (ρ = 0.48 across languages, p = 0.19; ρ = 0.25 per segment). We pre-registered the study, report the refuted hypothesis alongside the supported ones, and release the annotated parallel-span corpus with an independent re-derivation of every number.

PDF corpora advertise their size in tokens, but every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) is computed per document, and none decomposes its token total. Because PDF length is extremely skewed, the two units can describe the same corpus very differently. We ask how the headline statistics of a web-PDF corpus change when each document is weighted by the text it contributes rather than counted once. We used CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M Common Crawl PDFs, 32.6B tokens), the one public corpus that pairs the fragments Common Crawl stored with the re-fetched originals. Text mass is highly concentrated: 3.02% of text-bearing documents hold half the tokens (Gini 0.807). The clearest consequence is Common Crawl’s payload cap, which truncated 23.06% of these documents but 63.08% of their text. Reconstructing the truncated fragments and extracting both versions, two widely used text-layer parsers recover only 1.4% and 11.4% of that exposed text, so roughly 55–62% of the corpus’s text is unrecoverable from the crawl by such pipelines; under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be exposed. We recommend that corpus statistics be reported in both units, documents and tokens.

Corpus filtering strategies have evolved steadily, leading to increasingly effective data selection techniques that improve downstream LLM performance. However, while most filters focus on knowledge-rich and educational content, their effect on the linguistic and thematic composition of the documents in pre-training corpora remains underexplored. In this paper, we investigate how educational filtering affects both linguistic and thematic content by combining Propella-1 educational value annotations and the FinePDFs-Edu filter with the MultiCORE register (genre) framework. Focusing on the Finnish subset of the HPLT 4.0 corpus, we examine how educational filtering changes the distribution of registers and how educational value relates to register categories. By studying machine translated documents with high educational value we show that educational filters focus on content over textual quality. We further analyze changes in the thematic composition using Propella-1 business sector labels, examining whether educational filtering shifts the representation of different content domains. Finally, to assess the impact of filtering, we trained three small language models and evaluated their downstream performance on benchmark tasks. We find that educational filters disagree in their assessment of document educational value, show similar preferences for linguistic registers, but differ in the thematic content they select, resulting in different downstream performance across task types.

WiNCC is a reproducible and automatic construction pipeline for building an English–Nepali cross-lingual retrieval corpus that is difficulty-aware, relying on public Wikipedia and Wikidata. Part 1 reproduces a CLIRMatrix-style baseline. Nepali Wikipedia was crawled, articles resolved to a Wikidata QID, then intersected with Wikidata sitelinks to obtain Nepali–English query pairs and the corresponding English document corpus. These were sampled into gold pairs and padded with random English distractors to form a document baseline corpus. Part 2 adds a data-enrichment stage to Part 1 with multi-query, class-matched hard negatives. We compare the produced corpus against existing cross-lingual retrieval resources, run cross-lingual retrieval baselines with three distinct multilingual embedding models (BGE-M3, multilingual-E5-large, and Qwen3-Embedding-4B), and analyze slice-wise quality metadata and query-type-wise field metrics supporting the corpus’s use as a difficulty-aware benchmark. Through our corpus, we also find (1) a 27.8–41.5-point NDCG@10 drop from English to Nepali, (2) descriptions are 5–6× harder than titles, and (3) hard negatives are 2× stickier than random distractors.

The paper reports from a representativeness study covering Sámi languages in a Norwegian national web crawl. We extracted linguistic content from HTML in a considerable portion of the archive and applied language identification on paragraph-level. By aggregating the results on domain-name level, we were able to quantify the amount of Sámi language content in the web archive and identify contributing domains. Further, by comparing the URLs of the national crawl with a Sámi reference corpus, we identified domain names that were lacking in the national crawl and domains that lack depth. The results will be used to increase the coverage of Sámi language content in future web crawls and acquisitions.

Transformer-based web register classifiers are widely used to provide metadata for web crawls and to focus web-data sampling on specific register classes, such as news articles or advertisements. However, it remains unclear what these classifiers actually learn: do they detect pervasive linguistic features characteristic of registers, or do they rely on structural cues more indicative of genre? We examine this question through two hypotheses: the text-macrostructure hypothesis proposes that classifiers rely on structural features more typical of genres, whereas the feature-density hypothesis predicts that they identify typically register-associated features. Using English, Finnish, and French data from HPLT 3.0, we segment documents into thirds and compare classifier predictions for full documents and segments through five complementary analyses: main-label agreement, cosine similarity, confusion profiles, SACX keyword-weight correlations, and SACX keyword stability. As a more direct control, we compare predictions for a sample of untruncated documents with predictions for the same documents after their sentence order has been randomised. Top-label assignments and prediction vectors remain highly similar after shuffling, although the positional gradients are attenuated, indicating that sentence order contributes to classifier behaviour without determining its broader prediction profile. Overall, the results are more consistent with the feature-density hypothesis, although both hypotheses retain explanatory value.

Low-resource subsets of multilingual web corpora often carry large amounts of text in the wrong language, and audits keep rediscovering the problem in each new corpus release. Scoring one corpus at a time cannot say whether a later corpus repeats an earlier one’s errors on the same documents. We join mC4 and FineWeb-2, built independently from overlapping Common Crawl snapshots, at the document level for eight low-resource languages. A URL join and a shift-invariant fingerprint sort the first 250k FineWeb-2 documents per language into three overlap classes: co-selected (same page, near-identical text), page-shared (same page, divergent text), and unmatched. Overlap covers 11% to 74% of a FineWeb-2 subset, so both corpora draw heavily on the same pages, and the shared documents are clean anyway. Under a criterion that flags a document only when two classifiers reject its label, 76.5% of sampled mC4 Sundanese and 58.9% of mC4 Kyrgyz are flagged, against at most 2.7% of co-selected FineWeb-2 documents. We attribute the difference to selection at ingestion. Refiltering mC4 through GlotLID still leaves 0.7-17% of survivors disagreeing with fastText for six languages, and barely moves Sundanese or Yoruba. We release the fingerprints, the flags, and the code: https://github.com/garyzhang1006/wac13-lid-overlap

Pretraining corpus releases report how much duplication their pipelines removed, but we are not aware of one that estimates how much they missed. Measuring the residue is difficult because two recall problems apply at once: detectors miss duplicates they cannot see, and any tractable subsample misses duplicates whose partners were not sampled. In this paper we address both. We 1) run four detectors with distinct failure modes over a corpus and treat them as capture occasions, 2) recover what all four missed through capture-recapture, and 3) invert the sampling rate that produced the subsample, so that the result describes a full 15-trillion-token corpus rather than a convenient slice of it. The design is scored first against a planted population of known size, where it recovers 89.2% of the truth from a 44.5% detected rate while remaining a lower bound. Applied to the 14,868,862-document sample-10BT release of FineWeb, 4.50% of probes have a surviving near-duplicate, and capture-recapture raises that to 4.81% (95% CI 4.37-5.26). Inverting a sampling rate of one document in 1,532 then carries the figure to the corpus and gives a lower bound of 68 on the mean near-duplicate degree of a FineWeb document. Near-duplicate here means lexical overlap at Jaccard 0.35 or above. The bound is on a mean and not on a prevalence, since our design leaves the share of documents with any surviving duplicate anywhere between 7% and 73%. The survivors also carry a positional signature. The two copies of an exact-duplicate pair sit a mean 34.4% of the corpus apart against 33.3% expected under uniform sampling, which supports reading them as residue of FineWeb’s deliberate per-snapshot deduplication. A deduplicated corpus, therefore, is not a duplicate-free corpus, and the size of the gap can now be measured in one streaming pass.

This contribution uses the web as corpus to investigate metalinguistic discourses of the noun “female” between 2001 and 2019. It presents the construction of a diachronic dataset using search engines and compares results provided by Google, DuckDuckGo and Yandex, and, for Google, searches made with four different combinations of parameters. It then uses the resulting corpus to create a repertoire of arguments and ideologies used by online critics of “female”. Results show that, even though Google generated the most relevant results by far, supplementing it with other search engines was beneficial. They also suggest that, by 2015, the metalinguistic discourse of “female” had mostly solidified, with no new arguments emerging in the corpus, and that the debates are a response to new (at the time) and broader social tensions, specifically the rise of masculinist discourses in online spaces.

Web video makes massive spoken-language corpora possible, but automatic speech recognition (ASR) routinely normalizes away fillers, repetitions, repairs, and hesitations. Because Conversation Analysis (CA) transcripts meticulously document these interactional phenomena, they offer an ideal yet underutilized resource for ASR adaptation. We show that fine-tuning Whisper large-v3 on just 4.47 hours of CA-annotated Australian and US conversation dramatically improves verbatim transcription on heterogeneous web speech without sacrificing lexical accuracy. On a 57-clip benchmark from 39 Australian and New Zealand local-government channels, fine-tuning reduces verbatim word error rate from 16.06% to 11.39% (a 29% relative reduction) and raises filler recovery F1 from .117 to .819. These gains transfer directly to New Zealand speech absent from training (reducing verbatim error from 18.24% to 11.71%), demonstrating that small CA datasets teach the model a generalizable transcription policy rather than narrow acoustic memorization. We argue that targeted verbatim adaptation should become standard practice in web-as-corpus pipelines to ensure large-scale resources like CoANZSE preserve the interactional evidence needed for linguistic research.

Web registers, the situationally defined varieties of online text, are almost always annotated at document level. A document, however, may contain multiple such varieties. We label 80,000 documents from the HPLT web corpus at segment level in seven languages using the open-weight Gemma 4, reaching Cohen’s κ=0.73 against 1,878 human-annotated segments, comparable to previously reported document-level agreement. We find that most documents have two or more registers, and that they recur in similar positions and combinations across the studied languages. Yet on average, a single register accounts for 73% of the labeled text, and a quarter to a third of segments receive no label. Document-level labels are therefore mostly correct but incomplete. Multi-dimensional analysis shows that the segment-level labels correspond to linguistic differences. In all seven languages, registers are ordered the same way, from informational to involved text.

Web-crawled corpora now claim that they retain coverage of 1,000–2,000 languages, but what does such coverage mean for a language family at the extreme low-resource end? We audit the collective presence of the Muskogean family (Choctaw, Chickasaw, Muscogee, Alabama, Koasati, Mikasuki) across seven major web corpora. Only two of the six languages appear in any corpus, and their entire kept content is 28 documents comprising 16 distinct texts, dominated by 19th-century Bible translations; a single document comes from a present-day language community. We exhaustively audit both the kept documents and the 15,108 discarded ones, showing that (i) language identification (LID) assigns Muskogean labels to short non-Muskogean fragments with up to 99.7% persistence under re-identification, so quality filters, not LID, determine corpus content; (ii) those same filters discard about twice as many genuine Choctaw documents as they keep, including contemporary tribal-authored text; and (iii) four Muskogean languages are structurally invisible: no LID label exists, and verified Chickasaw text is silently absorbed into Choctaw. We release our audit annotations and seed-text URL list, and recommend family-level label sets, filter audits on the smallest subsets, and engagement with tribal data sovereignty for web corpus builders.

Web-crawled corpora now claim that they have coverage of more than a thousand languages, but their subsets for endangered languages remain very small. Prior audits examine what is inside these corpora. We ask a question that complements this idea: what happens to the endangered-language web content that does exist? We curate and verify an inventory of 40 pages from active institutional websites (tribal language programs, community dictionaries, and learning portals) for 16 endangered North American languages from 7 families, and we attribute losses along the full pipeline: (S1) whether Common Crawl ever crawled the page, (S2) whether language identification (LID) recognizes the captured text, (S3) whether a standard quality-filter cascade can help retain it, and (S4) whether any of it reaches a released corpus (FineWeb-2, GlotCC). Most of the loss happens before corpus construction even begins. These websites present to crawlers almost entirely in English, much of the actual language text is served by client-rendered dictionary applications that expose no text to crawlers, 6 of 16 languages have no label in the standard LID model, and none of the 36 inventory domains contributes a single document to any released per-language subset. We release the inventory, the pipeline, and the per-stage attribution to support both crawl-side and community-side remedies.

Questions scraped from the web are used across academia and industry as a proxy for what people want to know. Across QA training data, retrieval benchmarks, and content strategy, questions on a page are assumed to reflect human intent. We test this assumption at scale by extracting 13.4B question occurrences across 110 FineWeb snapshots (2013-2025), and report three findings. First, you can tell who is asking: provenance (the host/page of questions) leaves a signal in question form, and a logistic model can separate genuine user questions from templated/manufactured ones at AUC 0.725 via length and surrounding context rather than question type, though only 0.554 against commerce FAQ writing. Second, question frequency does not measure demand: the most-frequent questions are boilerplate/templated (over 70% of the top thousand), so occurrence counts measure how often a string was published and not how often it was asked. Third, over twelve years the genuine share of occurrences fell by 79% (42-56% after controlling for crawl composition), with question length and context decreasing. We present the first diachronic, occurrence-level measurement of web question provenance, and find the crawlable web’s questions have shifted from being asked by humans toward manufactured for machines to read.

This paper evaluates Wiktionary as an ethically crowdsourced lexicon for English dialects. We took a two-phase approach, providing an in-depth descriptive analysis of the crowdsourced lexicon for 12 national varieties of English before applying the lexicon to geo-referenced, country-level social media language data to examine the real-world performance of this crowdsourced dialect lexicon. We demonstrate that Wiktionary matches or exceeds the coverage of traditional dictionaries, such as the Oxford English Dictionary (OED), for regional and Outer-Circle varieties. Our dialect-specific case study on New Zealand English found high alignment between Wiktionary and the OED based on word-formation patterns (R = 0.883). Similarly, we observed high alignment between the dialect lexicon and geo-referenced social media language. While this paper found that Wiktionary has broad coverage of lexical properties, it also highlighted some of the macro-challenges involved in evaluating dialect-responsive language resources and tools, such as the role of language contact in dialects and register effects in web-based corpora.

Web-scale text corpora provide NLP researchers and corpus linguists with access to vast quantities of language data, but efficiently identifying the subsets of a large corpus that are relevant to a particular research aim remains a significant challenge. Existing approaches, including keyword search and task-specific classifiers, are often either too restrictive or depend on costly retraining for each new research question. We explore and evaluate the feasibility of a task-agnostic, LLM-driven descriptor schema for annotating web documents, using LLMs to generate natural-language labels that capture stylistic, tonal, topical, and structural properties of texts. Unlike prior work that emphasizes broad categorization or quality filtering, our approach yields fine-grained and interpretable descriptors designed to support flexible downstream applications. We evaluate the approach from multiple angles and show that descriptor-based annotation can enable flexible document retrieval while also exposing challenges related to schema harmonization and descriptor specificity.

Machine-generated text is becoming a routine part of the web. Most detectors, however, are still evaluated on curated datasets rather than real web content. We introduce PAWN++, a compact supervised detector that learns which likelihood signals from two frozen language models are most informative. On MAGE, our method reaches 98.36 AUROC in distribution and 98.42 under a joint domain and generator shift. It also transfers well to external benchmarks without target-specific training.

To test whether this robustness extends to the web, we construct WATTLE, a benchmark of 41,300 texts in which each machine output is tied to its original human web page. The resulting data capture realistic transformations of web content as well as variation across languages and time. We use WATTLE to compare our method with seven other detectors. Performance deteriorates sharply on realistic rewrites: our method reaches 70.1 AUROC overall and 58.1 on text produced by a frontier model. Likelihood-based detectors fall below chance, and no evaluated system identifies more than 12% of frontier rewrites at a 1% false-positive rate. These results expose a substantial gap between standard evaluation scores and reliable detection on the web.

Main-content extraction commonly precedes language identification (LID) and determines its input in web-corpus pipelines. We measure this dependency on 7,808 WebMainBench pages using three extractors and three LID systems. We compare human-selected main text with annotated-region and full-page extractor outputs. A context flip occurs when annotated-region extraction preserves the route on human-selected text but full-page extraction changes it. For pages with nonempty text and matching human-selected and annotated-region routes, context-flip rates are 1.10–3.20%. On the common eligible set, Magic-HTML has lower paired rates than Trafilatura for all three LID systems, with each corresponding 95% interval excluding zero. All-page route loss counts empty outputs or routes that differ from the route on human-selected text. Magic-HTML’s empty full-page outputs add 5.84 percentage points. The fixed snapshot’s coarse non_en–en difference is positive in all nine configurations and remains positive after standardization to the page-complexity field (meta.level). A 100-page audit finds 35 unsupported, 29 supported, and 36 indeterminate routes. Of 35 unsupported flips, 26 move from predicted English to another route, and 17 involve changed content selection alone or in mixed cases. Among supported flips, 21 move to English. Extraction context can change text availability and downstream routing.

Language models have achieved state-of-the-art performance across many natural language processing (NLP) tasks, but low-resource languages remain underrepresented due to limited pre-training data and evaluation benchmarks. While recent efforts have introduced corpora, models, and benchmarks for such languages, most focus on a single language in isolation. This is particularly limiting in multilingual countries such as Sri Lanka, where the lack of comparable multilingual resources hinders the development and fair evaluation of models across languages. To address this gap, we introduce CeyNews, the largest trilingual news corpus for Sinhala, Tamil, and English. The dataset contains over 1.02 million articles (136 million tokens) collected between 2013 and 2026 from three news sources, with full article text and rich metadata. Using CeyNews, we develop three downstream tasks: news source identification, news category classification, and headline generation, and evaluate multilingual models across different learning settings. Our results demonstrate the challenges of applying multilingual models to these tasks, particularly for the headline generation task. CeyNews fills a major gap in Sri Lankan and South Asian language resources and is publicly released to support research in multilingual NLP, LLM adaptation, and digital humanities.

Data selection algorithms for large language model (LLM) pretraining are designed to mine ‘quality’ from the web, but that notion is murky at best. This study explores whether sociolinguistic mobility – realized through language use, in context – offers a lens for understanding what such algorithms actually select. As a case study, the FineWeb pretraining pipeline is examined for how it reshapes two tangible axes of language use: the geographical provenance and register composition of texts drawn from ICE and GloWbE.

FineWeb rejection rates are highest, and fairly uniform across provenance groups, for web-native and high-context registers (90.9% and 99.2%, respectively) but within medium-context registers, association with the United States is moderately protective: the 7% rejection rate is well below the medium-context average of 38%, an effect corroborated by NPMI. Unlike prior work on learned, classifier-based filters and US-internal sociolect bias, this study finds comparable effects in a deterministic pipeline within the global spread of English. As pretraining data selection becomes increasingly opaque, these results suggest that algorithmic ‘quality’ does not represent a real-world range of English use manifested in the web-as-corpus – a gap made visible by tangible sociolinguistic dimensions.

Web video platforms contain naturally occurring audiovisual interaction, but their use as corpora requires explicit decisions about selection, metadata, annotation, privacy, and redistribution. We present a curated corpus of 256 YouTube videos of infants crying, covering vaccination, discomfort, fussiness, colic, hunger, birth, sleepiness, overstimulation, and other pain contexts. Candidate videos were identified through keyword searches and annotated from audiovisual evidence. To test what is lost when cry context is inferred from web text alone, we add a supporting analysis of the enTenTen21 web corpus. From 14,936 baby + CRY concordance lines, 1,816 cases were conservatively assigned to our ten shared cry-context labels; a complementary causative MAKE + infant/baby + CRY search revealed substantial formulaic and pragmatic noise. A close examination of Other Pain recovered supplementary distinctions such as teething/oral pain, circumcision, medical procedures, restraint, burns, and general injury. The comparison shows why retrieval evidence should remain distinct from audiovisual annotation. The corpus is a resource for modeling the cognitive and dialogic factors surrounding infant paralinguistic signals in caregiver and medical interaction. A pilot study on the vaccination videos in our audiovisual corpus illustrates how event-level annotation links cry pulses to procedures and adult responses.

Website owners increasingly restrict AI crawlers through robots.txt. Audits are English-centric and do not stratify results by language. We measure opt-out for 4,900 domains across fourteen languages and three resource tiers in FineWeb and FineWeb2. Among reachable domains, high-tier sites block 28.8 percent, versus 11.2 percent and 9.8 percent in the mid and low tiers; an exact language-partition permutation gives p = 0.008 for the high-low gap. The gradient is concentrated in naming: files naming any of fifteen AI agents fall from 33.6 percent to 10.6 percent across tiers, while named-agent blocking shows no tier ordering. Among files listing conventional crawlers, AI naming falls from 64.0 percent to 19.0 percent, consistent with vocabulary diffusion rather than maintenance failure. Applying current robots.txt retroactively to an earlier crawl, design-weighted document mass behind a Common Crawl opt-out is 31.2 percent for English; tier means are 26.5 percent, 8.9 percent, and 10.7 percent. Reachability falls from 78 percent in English to 39 percent in Uyghur. These results show a gradient in recorded policy that a consent-respecting rebuild would inherit. We release every fetched robots.txt, derived rates, and code.