I asked Cursor to “optimize SEO”, and ChatGPT still can’t read or recommend my site

Why “optimize SEO” does nothing

An AI coding tool does exactly what you ask. “Optimize SEO” names no page, no result and no test, so the tool picks the smallest change it knows, often a meta tag, and reports that it’s done. Nothing in the request tells it, or you, whether ChatGPT can now read the site.

Three more things work against a generic prompt:

• The usual causes aren’t on SEO checklists. When we checked a viral 20-point SEO list against Google’s own documentation, the three things that most often make a vibe-coded site invisible weren’t on it: text that exists only in JavaScript, soft 404s and open preview copies.
• The tool doesn’t see what the bots see. The major AI crawlers didn’t run JavaScript in Vercel’s analysis (Vercel, December 2024; third-party study), so a page can look finished in your browser and be nearly empty for them. Unless the prompt asks it to check the raw HTML, the tool may never find out.
• Some causes aren’t in your code. On 29 September 2026, katman.pro’s robots.txt allowed GPTBot while a Cloudflare setting returned 403 to AI training bots (own measurement). A change in your code doesn’t touch a setting in your CDN’s dashboard.

The rule from our book: never tell your coding tool to “improve rankings” or “optimize SEO”. Give it a measurable job (“a unique title on every page”) and an acceptance criterion (“no two pages share a title”).

Step 1: research before any code

An AI coding tool starts every session from zero, and where the rules aren’t written down, it guesses. So before the first prompt, write a short research.md:

• 10–12 customer sentences: what your customers would type into ChatGPT, without your brand name. Take them from sales calls, support emails and reviews.
• What assistants say today: ask each question logged out, with web search on, and note who is named and which pages are cited. The full question set is on How do I check whether ChatGPT recommends my site?.
• Your one sentence, in four parts: what the customer brings, what you do, what they get, which fear is unnecessary.
• A “not X” note, if assistants confuse you with another product.
• Competitors and proof, each with a source and a date.

Then start every session with prompt P0. This is a shortened version from the book:

We're working on SEO (Google, Bing) and GEO (visibility in assistants such as
ChatGPT, Claude, Gemini and Perplexity). First read research.md.
Rules:
- Add no number, customer review, competitor fact or claim that research.md
  doesn't source. Write [FILL] where you don't know.
- Put nothing in schema, the meta description or llms.txt that isn't visible
  on the page.
- Don't touch app screens behind a login; public pages only.
- After each change, list which file you changed and what.

Accept when: the tool summarises research.md and gets your OK on the page plan before it writes any code.

Next, ask for P1, an audit that changes nothing: a table of every public page (rendered on the server or in the browser, title, canonical, H1 count, JSON-LD, internal links pointing to it), plus whether a missing URL returns 404, whether menus use real <a href> links and whether a preview copy of the site is live. Fix from that report, not from a guess.

Step 2: one job per prompt, each with an acceptance check

Give the prompts in order, and don’t move on until the check passes. Three examples, shortened from the prompts in our book:

Text in the server HTML (P2)

The text of our public pages must be visible in the HTML without JavaScript:
headings, paragraphs, prices, FAQ answers and links. Pick the smallest change
for our framework (server rendering, static generation at build time, or
prerendering public pages) and explain why. Leave the app screens as they are.
Use real paths, not # fragments. When done, show that this command prints each
page's H1 and first paragraph:
curl -s https://DOMAIN/PAGE | sed 's/<[^>]*>/ /g' | tr -s ' ' | head -c 600

Accept when: with JavaScript off, every page’s H1 and first paragraph can be read.

Prove it:

curl -s https://yoursite.com/ | sed 's/<[^>]*>/ /g' | tr -s ' ' | head -c 600
curl -s https://yoursite.com/ | sed 's/<[^>]*>/ /g' | wc -w

The first command shows the opening text a bot receives; the second counts its words. Hundreds is good. A few dozen means your text is still in JavaScript.

robots.txt and the sitemap (P4)

Create robots.txt at the root:
- Allow search engines and AI bots by name: Googlebot, Bingbot, OAI-SearchBot,
  ChatGPT-User, GPTBot, ClaudeBot, Claude-User, Claude-SearchBot,
  PerplexityBot, Perplexity-User.
- Close only the app paths (for example /api/, /admin/, /app/), each ending
  in a slash.
- Last line: Sitemap: https://DOMAIN/sitemap.xml
Generate sitemap.xml from the page list at build time: only URLs that return
200, are canonical and should be indexed. Move lastmod only when a page's text
changes. Keep login, checkout and thank-you pages out of the sitemap and give
them noindex, but don't block a noindex page in robots.txt, or Google will
never see the noindex.

Accept when: robots.txt has no Disallow: /; every sitemap URL returns 200 and is canonical; no noindex page is in the sitemap.

Prove it:

curl -s https://yoursite.com/robots.txt | grep -iE '^disallow:[[:space:]]*/[[:space:]]*$' || echo "no site-wide block"
curl -s https://yoursite.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' |
  while read -r url; do echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"; done

Every line of the second command should start with 200. GPTBot and ClaudeBot collect training data, so allowing them is a business decision; OpenAI says blocking GPTBot doesn’t affect ChatGPT search. And robots.txt is only a request: if your CDN turns a bot away, the bot never reads it, which is why step 3 tests the bots directly.

JSON-LD from the page’s own data (P5)

Add JSON-LD to the pages, using only information visible on the page:
- site-wide: Organization (name, url, logo, sameAs) and WebSite;
- product or service pages: Service or SoftwareApplication, with exactly the
  price shown on the page;
- articles and guides: Article (headline, datePublished, dateModified, author);
- sub-pages: BreadcrumbList; pages with an FAQ section: FAQPage.
Generate the JSON-LD from the same data as the page content; never write it
a second time by hand. When done, add a test that checks every page's JSON-LD
block is valid JSON.

Accept when: Google’s Rich Results Test shows no errors, and every price and claim in the schema appears on the page.

Prove it (this needs python3), then paste the URL into the Rich Results Test:

curl -s https://yoursite.com/ | python3 -c "import sys,re,json; b=re.findall(r'application/ld\+json\">(.*?)</script', sys.stdin.read(), re.S); [json.loads(x) for x in b]; print(len(b), 'JSON-LD blocks are valid JSON')"

It should print a number above 0; an error means broken JSON. Schema isn’t a shortcut, though. It makes a page eligible for rich results without guaranteeing them, and Google says there’s no special schema.org structured data you need to add to appear in AI Overviews or AI Mode (Google, AI features and your website). Its job here is to say who you are in one machine-readable place that matches the page.

The other ten prompts

The rest of the prompt book follows the same pattern: head tags from one source (P3), situation pages (P6), an honest comparison guide (P7), llms.txt and the not-X note (P8), images, internal links, 404s and preview URLs, speed, measurement, and a final check before every publish (P14). The job and acceptance criterion of each one are on the Katman method; the full prompts are in the Katman book.

Step 3: verify with commands, not with “done”

Don’t take “done” from your coding tool on trust, and run the checks again after every technical fix: one fix can quietly break another. On our sample site, a Cache-Control: no-transform header switched off HTML compression, and pages went out about 4 times larger (own measurement).

SITE=https://yoursite.com
echo "404:   $(curl -s -o /dev/null -w '%{http_code}' $SITE/this-page-does-not-exist)"
echo "words: $(curl -s $SITE/ | sed 's/<[^>]*>/ /g' | wc -w)"
for b in GPTBot/1.2 OAI-SearchBot/1.0 ClaudeBot/1.0 PerplexityBot/1.0; do
  echo "$b: $(curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 (compatible; $b)" $SITE/)"
done
curl -s -o /dev/null -D - -H 'Accept-Encoding: br, gzip' $SITE/ | grep -i content-encoding

What good looks like:

• missing page: 404;
• words: hundreds, not dozens;
• every bot: 200;
• last line: br or gzip.

A bot that gets a 403 is usually being turned away by your CDN or firewall, not by your code: see Cloudflare is blocking AI crawlers, or run the AI crawler checker. The six one-minute tests for vibe-coded sites are on My vibe-coded site is invisible to ChatGPT. When everything passes, ask your coding tool to save these checks as a script that runs before every deploy: that’s prompt P14.

Passing these checks shows that bots can read the site; it doesn’t show whether assistants recommend it. OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search. Ask your fixed question set again 14 days after each change and compare: How do I check whether ChatGPT recommends my site?

Where Katman fits

Katman is an MCP server that runs inside Cursor, Claude Code, VS Code and other AI coding tools, so the research, the prompts and the checks happen next to your code. Katman keeps its work in the .katman/ folder of your project; your coding tool makes the code changes.

• Audit Week, $3 once for 7 days, finds what’s wrong and tells you what to do. katman_scan_code reads your codebase and reports whether pages render on the server or only in the browser; katman_audit_site checks the live site the way crawlers see it; katman_research writes research.md; katman_plan turns it into tasks, each with a prompt code, an acceptance criterion and the tool that verifies it. It doesn’t build pages or write files in your site.
• Katman Pro, $29 a month at the founding price for the first 100 customers (then $49), or $290 a year, adds the tools that do the work: katman_prompt returns P0–P14 filled with your research, with notes for your framework; katman_page_brief writes page briefs; katman_generate produces robots.txt, llms.txt and JSON-LD and writes them into your project only when you pass write: true.
• The PDF course, the Katman book and the 30-day GEO workbook, sold separately from Audit Week and Pro, has the whole prompt book: all 15 prompts with their acceptance criteria, if you’d rather paste them yourself. See the PDF course.

More on what Katman does and on pricing.

Sources: the Katman book (29 September 2026); Google, AI features and your website; OpenAI crawlers; Vercel’s analysis linked inline (third-party study). Google and OpenAI pages checked 30 September 2026. Written by the Katman team, Rnify Limited.