katman / CASE STUDY

CASE STUDY · KATMAN-AUDIT v0.1.0

Cloudflare blocks AI crawlers by default — our own audit caught it on our own site

We build a tool that measures AI answer-engine visibility. On the day it was finished, we ran it on our own domain — and it came back with a CRITICAL finding against us. This page is the honest record of what happened, with evidence.

What happened

katman.pro is served behind a Cloudflare Worker. On 2026-08-30 we finished the first version of katman-audit and, following our own advice, pointed it at our own domain before pointing it anywhere else.

The second check came back red: "AI crawler policy — FAIL. Blocking: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider." A visibility product whose own site was unreadable by the very models it optimizes for.

The cause was not our robots.txt. Our file allows everything. The blocking rules were injected above it by Cloudflare's "managed robots.txt" feature (content-signals), which adds Disallow: / rules for known AI training crawlers — unless you explicitly turn it off in the dashboard.

The evidence

This is a verbatim excerpt of what https://katman.pro/robots.txt served before the fix — note the managed block above our own rules:

User-agent: ClaudeBot / Disallow: / · User-agent: GPTBot / Disallow: / · User-agent: Google-Extended / Disallow: / · User-agent: CCBot / Disallow: / · User-agent: Bytespider / Disallow: / — followed by "# BEGIN Cloudflare Managed content".

The cruelest part: the blocking groups take precedence over our permissive rules for those user-agents, so no amount of editing our own robots.txt could have fixed it. The toggle lives in the Cloudflare dashboard (AI Crawl Control), not in your repo.

The fix and the re-test

One toggle in the Cloudflare dashboard (AI Crawl Control → Managed robots.txt: off). The managed block disappeared from robots.txt within minutes.

Re-running katman-audit on ourselves: 0 critical, 0 warnings — every check green. The exit code went from 2 (critical) to 0. Both runs are dated 2026-08-30.

Total cost of the incident: roughly one day of AI-crawler invisibility that we know of, and it would still be running if we had not audited ourselves first.

Why this matters to you

Many hosts and CDNs now ship AI-blocking defaults. If nobody runs this check, the site's answer-engine visibility is zero while every other SEO investment continues — and nothing in Google Analytics or Search Console will tell you.

If a visibility product can miss this on its own domain, any site can. The check costs less than a minute of crawling; do it before you spend a euro on content.

Second lesson from the same incident: our own analytics could not see our AI channel for weeks either, because ChatGPT sends no referrer and the traffic hides inside "direct". Measurement blindness compounds quietly.