---
title: "Astro SEO for AI search: static HTML, sitemap, llms.txt"
description: "Astro ships plain HTML by default, so AI crawlers can read it. Still on you: the sitemap integration, robots.txt, llms.txt, JSON-LD and client:only islands."
url: https://katman.pro/astro-seo-ai-search
lang: en
updated: 2026-10-22
---

Guide Updated 22 Oct 2026

# Is Astro good for SEO and AI search, and what still needs setting up?

> Astro is a good fit for AI search because its default static output is plain HTML that crawlers read without running JavaScript. What it doesn’t do for you: set site and add the sitemap integration, write robots.txt and llms.txt, give each page its title, description, canonical and JSON-LD, return a real 404, and keep important text out of client:only islands. katman.pro runs on Astro; this page shows how we set it up.

## Why Astro starts ahead

Astro’s `output` option defaults to `'static'`: every page is prerendered at build time ([Astro configuration reference](https://docs.astro.build/en/reference/configuration-reference/), read 15 Oct 2026). Components without a `client:*` directive render to HTML “without JavaScript”. Crawlers that don’t run scripts, which in Vercel and MERJ’s 2024 log study included OpenAI’s and Anthropic’s ([Do AI crawlers render JavaScript?](https://katman.pro/do-ai-crawlers-render-javascript)), get the full text.

The exception is `client:only`, which “skips HTML server rendering, and renders only on the client” ([Astro directives reference](https://docs.astro.build/en/reference/directives-reference/), read 15 Oct 2026). A React SPA dropped into Astro as one `client:only` island is still an empty div to those bots.

## Set it up in six steps

1.  **Set `site` and add the sitemap.** `npx astro add sitemap` installs @astrojs/sitemap (3.7.4 on npm, 15 Oct 2026); the build writes `sitemap-index.xml` and `sitemap-0.xml`. Use `filter` to drop thank-you and login pages.

```js
// astro.config.mjs
import { defineConfig } from "astro/config";
import sitemap from "@astrojs/sitemap";

export default defineConfig({
  site: "https://yoursite.com",
  integrations: [sitemap({ filter: (page) => !page.includes("/thanks") })],
});
```

2.  **Write robots.txt** in `public/robots.txt`, or as an endpoint so it reads `site`:

```ts
// src/pages/robots.txt.ts
import type { APIRoute } from "astro";

const bots = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot", "Claude-User", "ClaudeBot", "PerplexityBot"];

export const GET: APIRoute = ({ site }) =>
  new Response(
    ["*", ...bots].map((b) => `User-agent: ${b}\nAllow: /\nDisallow: /app/\n`).join("\n") +
      `\nSitemap: ${new URL("sitemap-index.xml", site)}\n`,
  );
```

3.  **Generate llms.txt from your content**, so it can’t drift from the pages:

```ts
// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { getCollection } from "astro:content";

export const GET: APIRoute = async ({ site }) => {
  const pages = await getCollection("pages");
  const lines = pages.map((p) => `- [${p.data.title}](${new URL(p.id, site)}): ${p.data.description}`);
  return new Response(`# Yoursite\n\n> One sentence on what you do.\n\n## Pages\n\n${lines.join("\n")}\n`);
};
```

4.  **Head tags in one layout.** Title, meta description and `<link rel="canonical" href={new URL(Astro.url.pathname, Astro.site)} />` from each page’s props.
5.  **JSON-LD** with `<script type="application/ld+json" set:html={JSON.stringify(schema)} />`, stating the same facts as the visible page.
6.  **A real 404.** `src/pages/404.astro` builds `404.html`; check that your host serves it with a 404 status, not a 200.

## How katman.pro does it

This site is an Astro 7 static build served by a Cloudflare Worker. Three choices differ from the defaults:

-   **Sitemap from an endpoint, not the integration,** so each `lastmod` is the day a page’s words last changed and scheduled pages stay out until their time.
-   **robots.txt with one group per AI bot,** generated from a list of crawler tokens, plus a pointer to llms.txt.
-   **Markdown twins.** The Worker answers `Accept: text/markdown` with the page’s Markdown, while the HTML page stays canonical.

Example, our own measurement on 15 Oct 2026:

```text
$ curl -s -A "...GPTBot/1.4..." https://katman.pro/method | sed 's/<[^>]*>/ /g' | wc -w
3564
$ curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -H 'Accept: text/markdown' https://katman.pro/method
200 text/markdown; charset=utf-8
$ curl -s -o /dev/null -w '%{http_code}\n' https://katman.pro/this-page-does-not-exist
404
```

## Where Astro sites still go wrong

-   **Text inside `client:only` islands** or fetched by an island after load. Render it in the `.astro` page instead.
-   **Server islands** (`server:defer`) render “outside the scope of the rest of the page”, in a later request; keep key text out of them.
-   **Dynamic routes in `output: 'server'`** don’t reach the sitemap integration; list them in your own sitemap endpoint.
-   **The CDN says no.** Astro’s HTML doesn’t help if Cloudflare answers AI bots with a 403: see [Cloudflare blocking AI crawlers](https://katman.pro/cloudflare-blocking-ai-crawlers).

## Astro or Next.js for a site that needs to be found

|  | Astro | Next.js App Router |
| --- | --- | --- |
| Default output | Static HTML, no JavaScript unless an island asks for it | Server-rendered or prerendered HTML plus the React runtime |
| Sitemap | `@astrojs/sitemap` integration or your own endpoint | `app/sitemap.ts` |
| robots.txt, llms.txt | `public/` or `src/pages/*.ts` endpoints | `app/robots.ts`; llms.txt in `public/` ([details](https://katman.pro/nextjs-seo-ai-search)) |
| Typical way to lose text | `client:only` islands | Text fetched in `useEffect`; streamed metadata |

To a crawler, both can deliver the same complete page. Pick by what the rest of the product needs, not by SEO.

Moving from React? If only the marketing pages need to be found, a common split is Astro for public pages and the existing SPA under `/app` ([React SPA options](https://katman.pro/react-spa-seo-ai-crawlers)). For llms.txt expectations, see [What is llms.txt?](https://katman.pro/what-is-llms-txt).

## Where Katman fits, and where it doesn’t

Katman is an MCP server inside your AI coding tool. `katman_scan_code` recognizes Astro and reports which pages ship their text without JavaScript; `katman_generate` writes robots.txt, llms.txt and JSON-LD from one research file, plus a server-side bot counter for Astro; `katman_audit_site` checks the live pages as crawlers fetch them. Your coding tool applies the changes. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped ([what the MCP does](https://katman.pro/mcp)). A clean Astro build is readable; whether an assistant recommends you depends on the pages you write, which [the method](https://katman.pro/method) covers, and nobody can promise it.

Written by the Katman team, Rnify Limited. Astro docs and npm read 15 October 2026; katman.pro measured the same day.

## Who this is not for

-   Astro sites behind a login: keep them out of search.
-   Teams choosing a framework for a web app: this page covers the public, content side.
-   Starlight docs sites that already ship a sitemap: check what your theme generates before adding another.

## Frequently asked questions

### Is Astro better than Next.js for SEO?

Not by default in what a crawler receives: both can send complete HTML. Astro makes static HTML the default and ships no JavaScript unless you ask for it, so there are fewer ways to break it.

### Do I need @astrojs/sitemap?

It is the quickest way: npx astro add sitemap, set site, and the build writes sitemap-index.xml. Write your own endpoint instead if you want per-page lastmod dates or need dynamic routes rendered on demand, which the integration can’t list.

### Where does llms.txt go in Astro?

Either public/llms.txt for a hand-written file, or src/pages/llms.txt.ts, an endpoint that Astro turns into a static file at build time in static output.

### Can AI crawlers read my React or Vue islands in Astro?

Islands with client:load or client:visible are rendered to HTML on the server and hydrated later, so their text is in the page. client:only skips server rendering, so its text isn’t.

### Does output: server hurt AI visibility?

No. On-demand pages are still rendered to HTML on the server. Only the sitemap integration loses track of dynamic routes, so list them yourself.

### How do I send IndexNow from an Astro site?

Astro has no built-in IndexNow. Host a key file in public/ and send changed URLs to the IndexNow API after each deploy, from CI or a small script.

## Related pages

-   [**Is React bad for SEO, or is it my single-page app?** React is fine for SEO; an empty SPA shell is not. Google renders JavaScript in a queue, the crawlers behind ChatGPT and Claude read only raw HTML. Test, then fix.](https://katman.pro/react-spa-seo-ai-crawlers)
-   [**How do I set up Next.js for AI search, and where does llms.txt go?** Next.js renders HTML on the server, yet AI crawlers can still get a blank page or a title in the body. Metadata, robots.ts, sitemap.ts and llms.txt in Next 16.](https://katman.pro/nextjs-seo-ai-search)
-   [**Do AI crawlers like GPTBot and ClaudeBot render JavaScript?** In Vercel and MERJ’s Dec 2024 log study, GPTBot, ClaudeBot and PerplexityBot didn’t run JavaScript; Google and Apple did. What’s measured, what’s assumed, a test.](https://katman.pro/do-ai-crawlers-render-javascript)
-   [**What is llms.txt, and is it worth writing?** llms.txt is a short Markdown summary of your site for language models. Google ignores it and most files get no requests; here is when it's still worth writing.](https://katman.pro/what-is-llms-txt)
-   [**Is Cloudflare blocking AI crawlers from my site?** robots.txt allows GPTBot but Cloudflare returns 403? The three AI bot types, the settings that block them, the 15 Sep 2026 change, a curl test and the fix.](https://katman.pro/cloudflare-blocking-ai-crawlers)
-   [**The Katman method: research, plan, build, verify** Four steps, three gates and 15 prompts with acceptance criteria that make a site readable, quotable and recommendable by ChatGPT, Claude, Gemini and Perplexity.](https://katman.pro/method)
-   [**Katman MCP** Katman MCP runs in Cursor, Claude Code and VS Code: it scans your code, audits your live site like AI crawlers do and checks every fix. Katman Pro from $99/month.](https://katman.pro/mcp)

Updated 22 October 2026 · Katman