{"id":41491,"date":"2026-06-22T16:27:10","date_gmt":"2026-06-22T14:27:10","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/anti-bot-waf-adoption-across-the-top-1000000-websites-open-dataset-cc-by-4-0-1m-rows-self-promotion\/"},"modified":"2026-06-22T16:27:10","modified_gmt":"2026-06-22T14:27:10","slug":"anti-bot-waf-adoption-across-the-top-1000000-websites-open-dataset-cc-by-4-0-1m-rows-self-promotion","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/anti-bot-waf-adoption-across-the-top-1000000-websites-open-dataset-cc-by-4-0-1m-rows-self-promotion\/","title":{"rendered":"Anti-bot \/ WAF Adoption Across The Top 1,000,000 Websites \u2014 Open Dataset (CC BY 4.0, ~1M Rows) [self-promotion]"},"content":{"rendered":"

<\/p>\n

\n

I scanned the Tranco top 1,000,000 sites (June 2026) and recorded, per domain, which anti-bot\/WAF vendor protects it and whether a plain request gets challenged. Releasing it as open data.<\/p>\n

– 998,497 probed, 818,614 reachable<\/p>\n

– Fields: domain, rank, reachable, protected, vendor(s), kind (waf\/captcha\/bot_management\/\u2026), difficulty band, block reason, enforcement, CAPTCHA type, final URL, status, probed_at \u2014 names only, no PII<\/p>\n

– Plus a top-50k “deep-page census” (86,792 rows) with a page_type field (homepage vs product\/listing\/profile)<\/p>\n

– License: CC BY 4.0<\/p>\n

Headline: 53.5% of reachable sites run a managed anti-bot\/WAF (Cloudflare ~45%), but only 9.8% actively challenged the request. The busiest sites run the least (top-1k 44% \u2192 long tail 54%).<\/p>\n

Dataset (gzipped JSONL + sample + summary.json): https:\/\/github.com\/Crawlora-org\/anti-bot-adoption-index-data<\/a><\/p>\n

Open-source detector CLI: go install github.com\/Crawlora-org\/crawlora-antibot@latest<\/a><\/p>\n<\/div>\n

submitted by \/u\/the_bigbang <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>