Is Astro good for SEO and AI search, and what still needs setting up?

Why Astro starts ahead

Astro’s output option defaults to 'static': every page is prerendered at build time (Astro configuration reference, read 15 Oct 2026). Components without a client:* directive render to HTML “without JavaScript”. Crawlers that don’t run scripts, which in Vercel and MERJ’s 2024 log study included OpenAI’s and Anthropic’s (Do AI crawlers render JavaScript?), get the full text.

The exception is client:only, which “skips HTML server rendering, and renders only on the client” (Astro directives reference, read 15 Oct 2026). A React SPA dropped into Astro as one client:only island is still an empty div to those bots.

Set it up in six steps

1. Set site and add the sitemap. npx astro add sitemap installs @astrojs/sitemap (3.7.4 on npm, 15 Oct 2026); the build writes sitemap-index.xml and sitemap-0.xml. Use filter to drop thank-you and login pages.

// astro.config.mjs
import { defineConfig } from "astro/config";
import sitemap from "@astrojs/sitemap";

export default defineConfig({
  site: "https://yoursite.com",
  integrations: [sitemap({ filter: (page) => !page.includes("/thanks") })],
});

2. Write robots.txt in public/robots.txt, or as an endpoint so it reads site:

// src/pages/robots.txt.ts
import type { APIRoute } from "astro";

const bots = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot", "Claude-User", "ClaudeBot", "PerplexityBot"];

export const GET: APIRoute = ({ site }) =>
  new Response(
    ["*", ...bots].map((b) => `User-agent: ${b}\nAllow: /\nDisallow: /app/\n`).join("\n") +
      `\nSitemap: ${new URL("sitemap-index.xml", site)}\n`,
  );

3. Generate llms.txt from your content, so it can’t drift from the pages:

// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { getCollection } from "astro:content";

export const GET: APIRoute = async ({ site }) => {
  const pages = await getCollection("pages");
  const lines = pages.map((p) => `- [${p.data.title}](${new URL(p.id, site)}): ${p.data.description}`);
  return new Response(`# Yoursite\n\n> One sentence on what you do.\n\n## Pages\n\n${lines.join("\n")}\n`);
};

4. Head tags in one layout. Title, meta description and <link rel="canonical" href={new URL(Astro.url.pathname, Astro.site)} /> from each page’s props.
5. JSON-LD with <script type="application/ld+json" set:html={JSON.stringify(schema)} />, stating the same facts as the visible page.
6. A real 404. src/pages/404.astro builds 404.html; check that your host serves it with a 404 status, not a 200.

How katman.pro does it

This site is an Astro 7 static build served by a Cloudflare Worker. Three choices differ from the defaults:

• Sitemap from an endpoint, not the integration, so each lastmod is the day a page’s words last changed and scheduled pages stay out until their time.
• robots.txt with one group per AI bot, generated from a list of crawler tokens, plus a pointer to llms.txt.
• Markdown twins. The Worker answers Accept: text/markdown with the page’s Markdown, while the HTML page stays canonical.

Example, our own measurement on 15 Oct 2026:

$ curl -s -A "...GPTBot/1.4..." https://katman.pro/method | sed 's/<[^>]*>/ /g' | wc -w
3564
$ curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -H 'Accept: text/markdown' https://katman.pro/method
200 text/markdown; charset=utf-8
$ curl -s -o /dev/null -w '%{http_code}\n' https://katman.pro/this-page-does-not-exist
404

Where Astro sites still go wrong

• Text inside client:only islands or fetched by an island after load. Render it in the .astro page instead.
• Server islands (server:defer) render “outside the scope of the rest of the page”, in a later request; keep key text out of them.
• Dynamic routes in output: 'server' don’t reach the sitemap integration; list them in your own sitemap endpoint.
• The CDN says no. Astro’s HTML doesn’t help if Cloudflare answers AI bots with a 403: see Cloudflare blocking AI crawlers.

Astro or Next.js for a site that needs to be found

To a crawler, both can deliver the same complete page. Pick by what the rest of the product needs, not by SEO.

Moving from React? If only the marketing pages need to be found, a common split is Astro for public pages and the existing SPA under /app (React SPA options). For llms.txt expectations, see What is llms.txt?.

Where Katman fits, and where it doesn’t

Katman is an MCP server inside your AI coding tool. katman_scan_code recognizes Astro and reports which pages ship their text without JavaScript; katman_generate writes robots.txt, llms.txt and JSON-LD from one research file, plus a server-side bot counter for Astro; katman_audit_site checks the live pages as crawlers fetch them. Your coding tool applies the changes. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped (what the MCP does). A clean Astro build is readable; whether an assistant recommends you depends on the pages you write, which the method covers, and nobody can promise it.

Written by the Katman team, Rnify Limited. Astro docs and npm read 15 October 2026; katman.pro measured the same day.