---
title: "AI crawler checker: can ChatGPT's bots reach your site?"
description: "Free: we load your homepage as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 8 other bots, then show which ones your CDN blocks and what robots.txt says."
url: https://katman.pro/tools/ai-crawler-checker
lang: en
updated: 2026-09-30
---

Free tool · AI crawler checker

# Can ChatGPT's bots *reach your site*?

Enter your site's address. We request your homepage once as each of 12 crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, and once as a normal browser, then show which ones your CDN or firewall turns away and what your robots.txt allows. Free, no sign-up.

## How to read the result

HTTP status

What your server or CDN answered to that user-agent. 200 means the page came back.

Blocked?

Blocked means 401, 403, 429 or 503, a Cloudflare challenge header (cf-mitigated), or a challenge page such as “Just a moment…” or “Attention Required”.

robots.txt for /

Whether your robots.txt allows that bot to fetch the homepage, and which group decided: a group that names the bot, or the \* group.

Baseline

The same request as a normal browser. If the browser is blocked too, the result says more about your firewall than about AI bots.

## Three kinds of AI bots, three different costs of blocking

| Kind | What it does | Bots | If you block it |
| --- | --- | --- | --- |
| Training | Collects pages to train models. | GPTBot, ClaudeBot, CCBot, Bytespider | Future models cannot learn about you from your site. According to OpenAI, blocking GPTBot does not affect ChatGPT search; each setting is independent. |
| Search | Fills the assistant’s search index; the pages shown in an answer are picked from it. | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You do not appear in that assistant’s search. OpenAI says a site that blocks OAI-SearchBot is not shown in ChatGPT search answers. |
| User (agent) | Opens a page during a chat because the user asked for it. | ChatGPT-User, Claude-User, Perplexity-User | When someone gives the assistant your link, it cannot read your page. OpenAI notes that robots.txt may not apply to this bot. |

Google-Extended and Applebot-Extended are not crawlers but robots.txt permission names, so testing them with a user-agent shows nothing. Googlebot, Bingbot and Applebot are used for both search and training.

## Fixing it on Cloudflare

1.  Cloudflare dashboard → your domain → Security → Settings: turn off “Block AI bots”.
2.  In the same place, AI bot policies: allow Search and Agent. Decide about Training on purpose.
3.  Turn off Managed robots.txt, AI Labyrinth and, if it turns bots away, Bot Fight Mode. Turn off Email Address Obfuscation so bots can read your address as text.
4.  Run this check again. OpenAI says changes take about 24 hours to show up on their side.
5.  For developers, the same settings are in the API: PUT /zones/<zone>/bot\_management with ai\_bots\_protection set to "disabled" and ai\_training set to your decision.

Rule

robots.txt is a request; the CDN’s decision is the door. If a bot gets a 403, it no longer matters what your robots.txt says.

## Check it yourself

The same test in a terminal on macOS or Linux. A user-agent test only shows rules that look at the user-agent; Google checks Googlebot by IP, so look at Search Console’s crawl stats for Googlebot.

Terminal

```
for b in GPTBot/1.2 ClaudeBot/1.0 CCBot/2.0 OAI-SearchBot/1.0 \
         Claude-SearchBot/1.0 PerplexityBot/1.0 ChatGPT-User/1.0 Claude-User/1.0; do
  printf "%-22s %s\n" "$b" "$(curl -s -o /dev/null -w '%{http_code}' \
    -A "Mozilla/5.0 (compatible; $b)" https://your-site.com/)"
done
```

## Bots are only the first gate

Getting in is gate one. Katman MCP runs this test inside a full audit of your live site (text without JavaScript, real 404s, canonicals, sitemap, JSON-LD) and tells your AI coding tool what to fix. Audit Week unlocks the audit for 7 days, for $3 once.

[Start Audit Week: $3](https://katman.pro/app/login?next=%2Fapp%2Fbilling%3Fbuy%3Daudit_week) [See Katman MCP](https://katman.pro/mcp)

## Frequently asked questions

### Why does my robots.txt say allow but the bot still gets 403?

Because your CDN or firewall answers before robots.txt is read. robots.txt is a request; the CDN’s decision is the door. On Cloudflare the usual cause is the old “Block AI bots” toggle or an AI training policy set to block.

### Is a 403 for GPTBot a problem if OAI-SearchBot gets in?

Not for ChatGPT search. OpenAI says each setting is independent, so blocking GPTBot only keeps your pages out of training data. Blocking OAI-SearchBot is the costly one: sites that block it are not shown in ChatGPT search answers.

### Can this tool test Googlebot for real?

No. We send Googlebot’s user-agent string, but Google verifies Googlebot by IP address, so a firewall may treat the real crawler differently. Check Search Console → Settings → Crawl stats for the responses Googlebot really got.

### What do you store when I run a check?

The hostname you checked, the result (status per bot, CDN, robots.txt verdicts) and the time. For rate limiting we also keep a one-way hash of your IP address, which is deleted after 24 hours. We keep the results for 12 months and use them only for anonymous statistics about how many sites block AI crawlers.

### How often can I run it?

Ten checks per hour from one connection.

### My site blocks some bots on purpose. Is that wrong?

It is your decision. The book’s rule: search and user bots stay allowed, robots.txt and the CDN say the same thing, and the choice about training bots is deliberate, not a default you clicked past during setup.

## Related pages

-   [**Is Cloudflare blocking AI crawlers from my site?** robots.txt allows GPTBot but Cloudflare returns 403? The three AI bot types, the settings that block them, the 15 Sep 2026 change, a curl test and the fix.](https://katman.pro/cloudflare-blocking-ai-crawlers)
-   [**Katman MCP** Katman MCP runs in Cursor, Claude Code and VS Code: it scans your code, audits your live site like AI crawlers do and checks every fix. Audit Week: $3, 7 days.](https://katman.pro/mcp)
-   [**What is GEO, and how does a site get named by ChatGPT?** GEO means getting ChatGPT, Claude, Gemini and Perplexity to read, quote and recommend your site: how it differs from SEO and AEO, and how to measure it.](https://katman.pro/what-is-geo)

Updated 30 September 2026 · Katman