Austin Cheng

Author directory

2026

Website owners increasingly restrict AI crawlers through robots.txt. Audits are English-centric and do not stratify results by language. We measure opt-out for 4,900 domains across fourteen languages and three resource tiers in FineWeb and FineWeb2. Among reachable domains, high-tier sites block 28.8 percent, versus 11.2 percent and 9.8 percent in the mid and low tiers; an exact language-partition permutation gives p = 0.008 for the high-low gap. The gradient is concentrated in naming: files naming any of fifteen AI agents fall from 33.6 percent to 10.6 percent across tiers, while named-agent blocking shows no tier ordering. Among files listing conventional crawlers, AI naming falls from 64.0 percent to 19.0 percent, consistent with vocabulary diffusion rather than maintenance failure. Applying current robots.txt retroactively to an earlier crawl, design-weighted document mass behind a Common Crawl opt-out is 31.2 percent for English; tier means are 26.5 percent, 8.9 percent, and 10.7 percent. Reachability falls from 78 percent in English to 39 percent in Uyghur. These results show a gradient in recorded policy that a consent-respecting rebuild would inherit. We release every fetched robots.txt, derived rates, and code.