---
title: "Cloudflare blocking AI crawlers? How to test and fix it"
description: "robots.txt allows GPTBot but Cloudflare returns 403? The three AI bot types, the settings that block them, the 15 Sep 2026 change, a curl test and the fix."
url: https://katman.pro/cloudflare-blocking-ai-crawlers
lang: en
updated: 2026-09-30
---

Situation Updated 30 Sep 2026

# Is Cloudflare blocking AI crawlers from my site?

> If your site sits behind Cloudflare, AI crawlers can get a 403 even when your robots.txt welcomes them. Run one curl command per bot, switch off the old “Block AI bots” toggle and set the Search and Agent policies to Allow, and the bots get a 200 again; OpenAI says ChatGPT search picks up robots.txt changes in about 24 hours. You don’t have to allow model training to appear in ChatGPT search: OpenAI treats training and search as separate settings.

## The short version

Your robots.txt is a polite request. Cloudflare’s decision comes first: if a bot gets a 403 at the edge, nothing in your robots.txt matters. Test each AI bot with one command, then check four Cloudflare settings: the old “Block AI bots” toggle, the AI bot policies, managed robots.txt and Bot Fight Mode. This page follows chapter 6 of [our book](https://katman.pro/guide), plus one update from Cloudflare dated 15 September 2026. If every bot already gets a 200 and ChatGPT still doesn’t name you, the cause is elsewhere: see [Why doesn’t ChatGPT recommend my brand?](https://katman.pro/why-chatgpt-does-not-recommend-my-brand).

## Three kinds of AI bot

Blocking each kind has a different cost.

| Type | What it does | Bots | If you block it |
| --- | --- | --- | --- |
| Training | Collects pages to train models | GPTBot, ClaudeBot, CCBot (Common Crawl), Bytespider | Future models can’t learn about you from your site. OpenAI says blocking GPTBot doesn’t affect ChatGPT search. |
| Search | Fills the assistant’s search index, where answers pick their pages | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You don’t appear in that assistant’s search. OpenAI: sites that opt out of OAI-SearchBot aren’t shown in ChatGPT search answers. |
| User (agent) | Opens a page mid-chat because a user asked | ChatGPT-User, Claude-User, Perplexity-User | The assistant can’t read your page even when someone pastes your link. OpenAI notes robots.txt may not apply to this bot. |

Sources: [OpenAI crawlers](https://developers.openai.com/api/docs/bots), [Anthropic crawlers](https://support.claude.com/en/articles/8896518).

Google-Extended and Applebot-Extended are permission names in robots.txt, not crawlers. They have no user agent of their own, so testing them with curl shows nothing. Google says Google-Extended doesn’t affect Google Search, but it does control Gemini training and grounding in the Gemini app ([Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)). Googlebot, Bingbot and Applebot are “mixed-use”: one crawler for both search and training. That detail matters for the 15 September change below.

## The Cloudflare settings that affect AI bots

| Setting | What it does | Our recommendation |
| --- | --- | --- |
| Block AI bots (legacy; API `ai_bots_protection`) | Blocks verified training bots and some look-alikes: off, on every page, or only on pages with ads. Cloudflare marked it as deprecated on 15 September 2026. | Off. Set the policies below on purpose instead. |
| AI bot policies: Training, Search, Agent (API `ai_training`, `ai_search`, `ai_user`) | One choice per type: block everywhere, block on pages with ads, or allow. Training also has “Disallow AI Training” since 15 September 2026. | Search and Agent: Allow. Training: your business decision. |
| Managed robots.txt | Adds Cloudflare’s block to the top of your robots.txt: a no-training signal and `Disallow: /` for GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended, Amazonbot and meta-externalagent | Off. Write your own robots.txt. |
| AI Labyrinth | Hides links that lead rule-breaking crawlers into a maze of generated pages; it doesn’t block | Off: it adds nothing for GEO. |
| AI Crawl Control (formerly AI Audit) | Shows which AI bots come and how often. Blocking a bot here creates a WAF rule; the free plan recognises bots by user agent only | On, for read-only monitoring. Don’t block from here. |
| Bot Fight Mode | Challenges unverified bots; WAF rules can’t make exceptions | Off, if you see it stopping crawlers |
| Email Address Obfuscation | Hides email addresses behind JavaScript | Off, so bots and assistants can read your address |

Cloudflare’s own pages: [Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/), [managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), [AI Labyrinth](https://developers.cloudflare.com/bots/additional-configurations/ai-labyrinth/), [Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/).

## What changed on 15 September 2026

### What our book recorded

-   **July 2025:** Cloudflare started asking every newly added domain, during setup, whether to allow AI bots. If you clicked through setup quickly, you may not have noticed the question.
-   **15 September 2026 (announced 1 July 2026):** new domains start with Training and Agent bots blocked on pages with ads and Search bots allowed ([Cloudflare changelog](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)).
-   **Same announcement:** anyone blocking training, including through the old toggle, would also block crawlers that serve both search and training; Cloudflare named Googlebot, Applebot and Bingbot. Customers could opt out before 15 September.
-   **Announced, not confirmed.** When the book went to press on 29 September 2026, we hadn’t found a record confirming this part had taken effect. If it has, leaving the old toggle on could affect Google and Bing too.

### What Cloudflare’s own post says

Cloudflare’s blog post dated 15 September 2026 ([accountable mixed-use AI crawlers](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)) describes the change as live. “Block” and “Block on pages with ads” for Training now also apply to mixed-use crawlers, including Googlebot, Bingbot and Applebot, so either one affects search too. To refuse training and stay in search, Cloudflare points to the new “Disallow AI Training” setting, and older settings are migrated to the new controls. We haven’t yet confirmed the effect on a live zone ourselves. The practical rule: never choose “Block” for Training unless you accept disappearing from Google and Bing.

## It happened to us twice

### 30 August 2026: managed robots.txt

The first version of our audit tool flagged a critical issue on our own site: Cloudflare’s managed robots.txt was silently blocking GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider. We turned it off (own measurement).

### 29 September 2026: the old toggle

While writing the book we tested again. Our robots.txt allowed GPTBot, but the legacy “Block AI bots” toggle was on in “only on pages with ads” mode, and the training policy said “disallow”. katman.pro has no ads, yet training bots got a 403 on every page (own measurement, tested by user agent).

| Bot | Cloudflare category | Before | After |
| --- | --- | --- | --- |
| GPTBot, ClaudeBot, CCBot, Bytespider | AI Crawler (training) | 403 | 200 |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot | AI Search | 200 | 200 |
| ChatGPT-User, Claude-User, Perplexity-User | AI Assistant (user) | 200 | 200 |

The old toggle was the whole cause. When we switched it off, Cloudflare set the training policy to “disabled” by itself, so the “disallow” was only a trace of the toggle. Twenty seconds later, all ten bots got a 200.

### The Google side

In Search Console’s crawl stats for the last 90 days, 79% of Googlebot’s requests got a 200, 10% a 404 and 10% a 301, with no separate 403 line. But on 17 and 21 September 2026, 6 requests to pages that exist got “other 4xx” errors that our own code doesn’t send. The cause isn’t certain; the timing matches the 15 September change. Google fetches the pages normally in live tests, and we’re watching the crawl stats.

## How to test your site

User-agent spoofing only reveals rules that look at the user agent. You can’t imitate IP-verified bots like Googlebot; check Search Console for those.

```bash
# Status code per bot. 403 = blocked at the door.
for b in GPTBot/1.2 ClaudeBot/1.0 CCBot/2.0 OAI-SearchBot/1.0 Claude-SearchBot/1.0 \
         PerplexityBot/1.0 ChatGPT-User/1.0 Claude-User/1.0; do
  printf "%-22s %s\n" "$b" "$(curl -s -o /dev/null -w '%{http_code}' \
    -A "Mozilla/5.0 (compatible; $b)" https://yoursite.com/)"
done
```

Every line should say 200. The [AI crawler checker](https://katman.pro/tools/ai-crawler-checker) runs the same test from your browser.

### Where else to look

-   **Cloudflare → Security → Analytics → Events:** which request was blocked by which rule (on the free plan, a sample of the last 24 hours).
-   **AI Crawl Control → Metrics:** AI bot requests and their status codes.
-   **Search Console → Settings → Crawl stats → By response:** Googlebot’s 403 and “other 4xx” responses, with URL and date.
-   **Response headers:** `cf-mitigated: challenge` means Cloudflare served a challenge page instead of your content ([Cloudflare](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/)).
-   **Your server logs:** if bot names never appear, your door may be shut.

A CDN block is only one of six common traps; the other five are on [My vibe-coded site is invisible to ChatGPT](https://katman.pro/vibe-coded-site-invisible-to-chatgpt). If you use [Katman](https://katman.pro/mcp), `katman_audit_site` runs this bot test on every audit and names the Cloudflare setting to change.

## How to fix it

### In the dashboard

1.  Cloudflare → your domain → Security → Settings: switch off “Block AI bots”.
2.  In the same place, set the AI bot policies: Search and Agent to Allow. For Training, choose Allow, or “Disallow AI Training” if you want to refuse training but stay in Google and Bing.
3.  Switch off managed robots.txt, AI Labyrinth and Email Address Obfuscation, and Bot Fight Mode if it stops crawlers.
4.  Run the test again. OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search.

### With the API

The same settings live in your zone’s bot management config ([Cloudflare API](https://developers.cloudflare.com/api/resources/bot_management/)). Read it first and keep a copy:

```bash
curl -s "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/bot_management" \
  -H "Authorization: Bearer $CF_API_TOKEN"
```

Then switch off the legacy toggle and stop blocking search and user bots. Set `ai_training` to match your decision; in our zone, “disabled” is what the API showed once the policy stopped blocking.

```bash
curl -s -X PUT "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/bot_management" \
  -H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" \
  --data '{"ai_bots_protection":"disabled","ai_search":"disabled","ai_user":"disabled","ai_training":"disabled"}'
```

Run the GET again to confirm, then repeat the curl test. The API token needs permission to edit Bot Management.

## Should you allow training bots?

**If you allow them,** models can learn about you in your own words, and an assistant may know your name even when it answers without searching. Our sample site chose this: `ai-train=yes` in robots.txt and every Cloudflare block off. In its first week, ClaudeBot made 578 requests and GPTBot 362 (own measurement, 21–27 September 2026).

**If you block them,** your content stays out of training data, and you still appear in ChatGPT search. That can be a sensible choice for publishers and paid content.

Either way, three conditions hold: search and user bots stay open; robots.txt and the CDN say the same thing; and the decision is made on purpose, not by clicking “Next” during setup. We couldn’t measure a direct effect of training access on ChatGPT visitors: the sample site’s ChatGPT visitors came from pages the search bot had read.

Written by the Katman team, Rnify Limited. Cloudflare, OpenAI, Anthropic and Google pages checked 30 September 2026.

## Who this is not for

-   Sites that aren’t behind Cloudflare: the same test works on any CDN or firewall, but the settings have other names.
-   Owners who want to keep all AI use off their content: this page is about letting the right bots in.
-   Checking Googlebot or Bingbot access: they are verified by IP, so use Search Console and Bing Webmaster Tools, not curl.

## Frequently asked questions

### Why does Cloudflare block GPTBot when my robots.txt allows it?

Because Cloudflare’s rules run before any bot reads your robots.txt. robots.txt is a request; a 403 from the CDN is a locked door, and the bot never gets as far as your rules.

### Will blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search. OpenAI says GPTBot (training) and OAI-SearchBot (search) are independent settings, so keep OAI-SearchBot and ChatGPT-User allowed whatever you decide about training.

### Is it safe to turn off “Block AI bots”?

For visibility, yes: it only removes the AI-specific block and lets verified AI crawlers in. It doesn’t switch off Cloudflare’s other protections.

### What is “Disallow AI Training”?

A Training setting Cloudflare introduced on 15 September 2026. It publishes a no-training rule in your robots.txt while leaving mixed-use crawlers such as Googlebot free to crawl for search, according to Cloudflare’s post.

### How do I test Googlebot if I can’t fake its user agent?

Use Search Console. The URL inspection live test shows what Googlebot receives, and Settings → Crawl stats → By response lists 403 and other 4xx responses with dates.

### How long after the fix will ChatGPT see my pages?

OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search. OAI-SearchBot still has to crawl your pages after that, so look for its visits in your server logs.

### Should I block Google-Extended?

Only if you accept a cost. Google says Google-Extended doesn’t affect Google Search, but it does control whether your pages are used to ground answers in the Gemini app.

## Related pages

-   [**AI crawler checker** Free: we load your homepage as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 8 other bots, then show which ones your CDN blocks and what robots.txt says.](https://katman.pro/tools/ai-crawler-checker)
-   [**I built my site with Lovable, Bolt or Cursor, and ChatGPT never mentions it** Built it with Lovable, Bolt, v0 or Cursor and ChatGPT never mentions it? Six traps hide vibe-coded sites; each has a one-minute test and a fix for your AI tool.](https://katman.pro/vibe-coded-site-invisible-to-chatgpt)
-   [**Why doesn't ChatGPT recommend my brand?** ChatGPT names your competitors but never you? Check the seven causes in order, from blocked AI bots to missing situation pages, with a test and a fix for each.](https://katman.pro/why-chatgpt-does-not-recommend-my-brand)
-   [**The Katman method: research, plan, build, verify** Four steps, three gates and 15 prompts with acceptance criteria that make a site readable, quotable and recommendable by ChatGPT, Claude, Gemini and Perplexity.](https://katman.pro/method)
-   [**Glossary** Plain definitions of the terms behind AI visibility: GEO and AEO, AI crawlers, llms.txt, IndexNow, canonical, soft 404, situation pages and the Katman method.](https://katman.pro/glossary)
-   [**Katman MCP** Katman MCP runs in Cursor, Claude Code and VS Code: it scans your code, audits your live site like AI crawlers do and checks every fix. Katman Pro from $49/month.](https://katman.pro/mcp)

Updated 30 September 2026 · Katman