AI crawler user agents: the full list, with sources
AI companies run three kinds of bot: training crawlers (GPTBot, ClaudeBot, MistralAI-Training), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. This list gives each token, its purpose, whether it follows robots.txt and its published IP list, all read on the vendors’ own pages on 15 October 2026.
The short version
Three kinds of bot, and blocking each has a different cost. Training crawlers collect pages for model training. Search crawlers fill an assistant’s index, which is where its answers find pages to cite. User fetchers open a page because a person asked, and several vendors say robots.txt may not apply to them. Allow search and user bots if you want to be cited; training is a separate choice. Every row below was read on the vendor’s own page on 15 October 2026; vendors change these pages, so check the date.
The list
| Vendor | Token | Type | robots.txt | Published IPs |
|---|---|---|---|---|
| OpenAI | GPTBot | Training | Yes | gptbot.json |
| OpenAI | OAI-SearchBot | Search (ChatGPT) | Yes | searchbot.json |
| OpenAI | ChatGPT-User | User | May not apply | chatgpt-user.json |
| OpenAI | OAI-AdsBot | Ad landing pages only | n/a | adsbot.json |
| Anthropic | ClaudeBot | Training | Yes | bots.json (all three) |
| Anthropic | Claude-SearchBot | Search | Yes | same |
| Anthropic | Claude-User | User | Yes | same |
| Perplexity | PerplexityBot | Search (not training) | Yes | perplexitybot.json |
| Perplexity | Perplexity-User | User | Generally ignores | perplexity-user.json |
| Googlebot | Search | Yes | common-crawlers.json | |
| Google-Extended | Token only: Gemini training and grounding | Token | none (no crawler) | |
| GoogleOther | Generic, research | Yes | common-crawlers.json | |
| Google-CloudVertexBot | Vertex AI Agents, owner-requested | Yes | common-crawlers.json | |
| Google-Agent | User (agent) | Generally ignores | user-triggered-agents.json | |
| Apple | Applebot | Search, may train | Yes | applebot.json |
| Apple | Applebot-Extended | Token only: training opt-out | Token | none (no crawler) |
| Meta | meta-externalagent | Training, product indexing | Yes | not published |
| Meta | meta-webindexer | Search (Meta AI) | Yes | not published |
| Meta | meta-externalfetcher | User | May bypass | not published |
| Amazon | Amazonbot | Products, may train | Yes | ip-addresses |
| Amazon | Amzn-SearchBot | Search (Alexa), not training | Yes | searchbot-ip-addresses |
| Amazon | Amzn-User | User | May not follow all | live-ip-addresses |
| Common Crawl | CCBot | Open web archive | Yes | ccbot.json |
| Mistral | MistralAI-Training | Training | Yes | not published |
| Mistral | MistralAI-Index | Search (Vibe), not training | Yes | index ips |
| Mistral | MistralAI-User | User (Vibe) | Yes | user ips |
| DuckDuckGo | DuckAssistBot | AI-assisted answers, not training | Yes (72 hours) | duckassistbot.json |
| Microsoft | bingbot | Search | Yes | bingbot.json |
Sources, all read 15 Oct 2026: OpenAI, Anthropic, Perplexity, Google common crawlers, Google user-triggered fetchers, Apple, Meta, Amazon, Common Crawl, Mistral, DuckDuckGo, Bing.
Two rows need care. Amazonbot and meta-externalagent are not pure training bots: Amazon says Amazonbot improves its products and may train its models, and Meta says meta-externalagent may also index content for its products. Applebot feeds Spotlight, Siri and Safari search and may also train Apple’s models; Applebot-Extended is how you refuse the training part.
Exact user agent strings
As published by each vendor. W.X.Y.Z is the vendors’ own placeholder for a browser version that changes.
GPTBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Googlebot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
GoogleOther Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
Google-Agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36
Applebot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Amazonbot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Amzn-SearchBot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Amzn-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
CCBot CCBot/2.0 (https://commoncrawl.org/faq/)
MistralAI-Index Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
MistralAI-User Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
MistralAI-Training Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)
DuckAssistBot DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
bingbot Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
Anthropic publishes tokens but no full strings. Meta’s page gives its agents as meta-externalagent/1.1, meta-webindexer/1.1 and meta-externalfetcher/1.1, each followed by a link to its crawler page. Google and Bing also use mobile variants. The differences between three OpenAI bots are explained on GPTBot vs OAI-SearchBot vs ChatGPT-User.
Example: allow search and user bots, choose on training
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
The first group is search and user bots: keep it open. The second group is your training decision. To refuse training, change its Allow: / to Disallow: /; Googlebot, bingbot and Applebot keep crawling for search. Several user fetchers may skip robots.txt anyway, so the file is a statement of intent, and your CDN is the real gate.
Test it with curl
robots.txt can say yes while your firewall says no. Check what each bot gets:
for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User \
PerplexityBot Perplexity-User CCBot MistralAI-User DuckAssistBot; do
printf '%-18s %s\n' "$bot" "$(curl -s -o /dev/null -D - \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; $bot/1.0)" \
https://yoursite.com/ | head -n 1 | tr -d '\r')"
done
Every line should end in 200. A 403 usually means a CDN rule; Cloudflare is blocking AI crawlers covers the settings. This only tests rules that read the user agent: Googlebot, bingbot and Applebot are verified by IP or reverse DNS, so use Search Console and Bing Webmaster Tools for them. The free AI crawler checker runs the same test in your browser.
To confirm a log entry is genuine, match its IP against the vendor’s JSON file above. Anyone can type a bot’s name into a user agent.
What Katman does here, and what it can’t
Katman runs inside your AI coding tool. katman_audit_site reads your robots.txt rules for AI bots, tests bot access through your CDN, and names the line or setting to change; katman_generate writes a robots.txt from your training decision. Your coding tool applies it. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped. Or have us do it: Growth writes 30 pages a month for you.
It can’t send requests from vendors’ IP addresses, and opening the door is a precondition, not a result: nobody can promise an assistant will recommend you. Definitions are in the glossary.
Written by the Katman team, Rnify Limited. Every vendor page above checked 15 October 2026.
Who this is not for
- Blocking scrapers that lie about who they are: robots.txt and user agent rules only work on bots that identify themselves.
- SEO crawlers such as AhrefsBot or SemrushBot: this list covers AI companies and the search engines AI answers draw on.
- xAI (Grok): we found no crawler documentation from xAI, so it isn’t listed.
Frequently asked questions
What is the user agent string for ClaudeBot?
Anthropic doesn’t publish full user agent strings. Its help page names the three tokens (ClaudeBot, Claude-SearchBot, Claude-User) and links one IP list, claude.com/crawling/bots.json, for checking that a visit is really Anthropic’s.
Which AI bots ignore robots.txt?
By their own docs: Perplexity-User generally ignores it, ChatGPT-User and Amzn-User may not follow it, meta-externalfetcher may bypass it, and Google’s user-triggered fetchers such as Google-Agent generally ignore it. All are fetches a person asked for.
Is CCBot a training crawler?
CCBot builds Common Crawl’s open repository of web crawl data, which anyone can download and analyze, including for model training. Blocking it keeps your new pages out of future crawls, not out of any search engine.
Does Bing have a separate AI training token like Google-Extended?
Not on its crawler page. Bing lists bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview there, and names no AI training token, so keep bingbot allowed if you want to be in Bing.
Where are the official IP ranges for AI crawlers?
Each vendor publishes its own JSON file; the links are in the table on this page. Meta’s crawler page publishes none, and Mistral publishes none for MistralAI-Training. Refresh the lists often: vendors change them.
What is GoogleOther?
A generic Google crawler that product teams use for one-off fetches, such as internal research and development. Google says rules for GoogleOther don’t affect any specific product, including Search.