katman / Survey
DATED SURVEY · 2026-08-30
Who blocks AI crawlers?
We ran the open-source katman-audit engine against 22 major websites and recorded, per site, whether robots.txt blocks GPTBot, ClaudeBot and PerplexityBot, whether llms.txt exists, and whether a sitemap is declared. "Blocked" means a Disallow: / rule in a user-agent group matching the bot.
Headline: of 19 verifiable sites, 3 block at least one major AI crawler. llms.txt adoption: 9/22. Three sites could not be verified because their robots endpoint rejects non-browser agents — itself a finding.
| Site | GPTBot | ClaudeBot | PerplexityBot | llms.txt | Sitemap |
|---|---|---|---|---|---|
| openai.com | allowed | allowed | allowed | no | declared |
| anthropic.com | allowed | allowed | allowed | no | declared |
| github.com | allowed | allowed | allowed | yes | not declared |
| cloudflare.com | allowed | allowed | allowed | yes | declared |
| vercel.com | allowed | allowed | allowed | yes | declared |
| npmjs.com | n/a | n/a | n/a | no | n/a |
| stackoverflow.com | n/a | n/a | n/a | no | n/a |
| news.ycombinator.com | allowed | allowed | allowed | no | not declared |
| nytimes.com | blocked | blocked | blocked | no | declared |
| bbc.com | blocked | blocked | blocked | no | declared |
| theguardian.com | allowed | blocked | blocked | no | declared |
| wikipedia.org | allowed | allowed | allowed | no | declared |
| reddit.com | allowed | allowed | allowed | yes | not declared |
| hubspot.com | allowed | allowed | allowed | yes | not declared |
| shopify.com | allowed | allowed | allowed | yes | not declared |
| slack.com | allowed | allowed | allowed | yes | declared |
| notion.so | allowed | allowed | allowed | yes | declared |
| midjourney.com | n/a | n/a | n/a | no | n/a |
| perplexity.ai | allowed | allowed | allowed | no | declared |
| hepsiburada.com | allowed | allowed | allowed | no | declared |
| trendyol.com | allowed | allowed | allowed | yes | not declared |
| webrazzi.com | allowed | allowed | allowed | no | declared |
Method
Measured 2026-08-30 by fetching each site's robots.txt and /llms.txt the way an AI crawler would (single request per file, custom audit user-agent). "Blocked" requires an explicit site-wide Disallow in a user-agent group matching the bot. Sites whose robots endpoint returned an error are marked n/a. This page will be re-run and re-dated; crawler policies change frequently.
Caveats
robots.txt permission is not the whole story: some sites serve challenges to AI user-agents at the CDN level while their robots.txt stays permissive, and robots.txt "allowed" does not guarantee the content gets served. Treat this as one dated observation, not a verdict — exactly how we treat model answers.
Check your own site
The same engine that produced this table is open source — and if your host silently blocks AI crawlers, this is the fastest way to find out. Related: how it caught exactly that on our own site (our case study).
Related case study: Cloudflare blocks AI crawlers by default — how our own audit caught it