---
title: "Next.js SEO for AI search: metadata, robots, llms.txt"
description: "Next.js renders HTML on the server, yet AI crawlers can still get a blank page or a title in the body. Metadata, robots.ts, sitemap.ts and llms.txt in Next 16."
url: https://katman.pro/nextjs-seo-ai-search
lang: en
updated: 2026-10-20
---

Guide Updated 20 Oct 2026

# How do I set up Next.js for AI search, and where does llms.txt go?

> Next.js App Router pages render to HTML on the server, so most of the work is making sure nothing undoes that: text fetched in useEffect, ssr: false imports, and metadata that streams into the body for bots Next.js doesn’t recognize, such as GPTBot. Then add app/robots.ts, app/sitemap.ts, per-page metadata with a canonical, JSON-LD, and llms.txt as a static file in public/. Check each route with curl as an AI bot.

## Next.js already sends HTML; here is what undoes it

App Router pages are Server Components by default, and Next.js uses Client Components and the server payload “to prerender HTML” as well ([Next.js, “Server and Client Components”](https://nextjs.org/docs/app/getting-started/server-and-client-components), updated 5 Oct 2026, read 15 Oct 2026). A Next.js page is readable without JavaScript unless something takes the text out of the HTML:

-   **Data fetched in the browser.** A `"use client"` page that loads its copy in `useEffect` ships the loading state. Fetch in the Server Component and pass the data down as props.
-   **`next/dynamic` with `ssr: false`.** Whatever that component draws is missing from the HTML.
-   **Streaming metadata.** Since 15.2, when `generateMetadata` waits on dynamic data, Next.js sends the page first and appends the tags “to the `<body>` tag” later. Bots on its HTML-limited list get them in `<head>` instead ([generateMetadata](https://nextjs.org/docs/app/api-reference/functions/generate-metadata), updated 19 Aug 2026). The default list (in the Next.js repository, read 15 Oct 2026) includes Bingbot, applebot and Google’s special crawlers, but not GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot.

For that last one, keep metadata static or cached where you can. If it must be dynamic, extend the list. Your regex replaces the default one, so keep the bots you still need:

```ts
// next.config.ts
import type { NextConfig } from "next";

const config: NextConfig = {
  htmlLimitedBots:
    /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Bingbot|applebot|facebookexternalhit|Twitterbot|LinkedInBot|Slackbot/i,
};
export default config;
```

v0 produces Next.js App Router apps, so a v0 site follows the same rules ([our builder table](https://katman.pro/vibe-coded-site-invisible-to-chatgpt)).

## Metadata, robots.ts and sitemap.ts

Set the base URL and a title template once, then give every page its own title, description and canonical:

```tsx
// app/layout.tsx
export const metadata: Metadata = {
  metadataBase: new URL("https://yoursite.com"),
  title: { template: "%s | Yoursite", default: "Yoursite" },
};

// app/pricing/page.tsx
export const metadata: Metadata = {
  title: "Pricing",
  description: "Plans, limits and what each one includes.",
  alternates: { canonical: "/pricing" },
};
```

Robots rules per AI bot, with the app area closed:

```ts
// app/robots.ts
import type { MetadataRoute } from "next";

const aiBots = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot", "Claude-User", "ClaudeBot", "PerplexityBot", "Perplexity-User"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow: ["/app/", "/api/"] },
      { userAgent: aiBots, allow: "/", disallow: ["/app/", "/api/"] },
    ],
    sitemap: "https://yoursite.com/sitemap.xml",
  };
}
```

```ts
// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getPages } from "@/lib/pages";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const pages = await getPages();
  return pages.map((p) => ({ url: `https://yoursite.com${p.path}`, lastModified: p.updatedAt }));
}
```

Both files are route handlers that are cached by default ([robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots), updated 1 May 2026; [sitemap](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap), updated 18 Aug 2026). Use `lastModified` for the date the words changed, not the build date.

## Where llms.txt goes

Next.js has no `llms.txt` file convention. Two options:

| Option | Path | Watch out for |
| --- | --- | --- |
| Static file | `public/llms.txt`, served at `/llms.txt` | Hand-written files drift from your prices; generate it |
| Route handler | `app/llms.txt/route.ts` | `GET` handlers are dynamic by default since 15.0; with Cache Components on, the `dynamic` segment option is removed (16.0) |

The static file is the simpler of the two. Generate it before each build from the same data your pages use:

```js
// scripts/llms.mjs   (package.json: "prebuild": "node scripts/llms.mjs")
import { writeFileSync } from "node:fs";
import site from "../content/site.json" with { type: "json" };

const lines = [`# ${site.name}`, "", `> ${site.oneSentence}`, "", "## Pages", "",
  ...site.pages.map((p) => `- [${p.title}](https://yoursite.com${p.path}): ${p.summary}`)];
writeFileSync("public/llms.txt", lines.join("\n") + "\n");
```

Keep expectations low: Google ignores llms.txt, and most files get no requests at all ([What is llms.txt?](https://katman.pro/what-is-llms-txt)). It is cheap, and useful only when it says exactly what your pages say. A starting file comes out of the [llms.txt generator](https://katman.pro/llms-txt-generator).

## JSON-LD

Next.js recommends a plain `<script type="application/ld+json">` in the page, with `JSON.stringify(jsonLd).replace(/</g, "\\u003c")` to escape `<` ([Next.js, JSON-LD guide](https://nextjs.org/docs/app/guides/json-ld), updated 2 Mar 2026). Put the same facts in it that the visible page states.

## Check it as a bot (example)

```bash
UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'
URL=https://yoursite.com/pricing
curl -s -A "$UA" "$URL" | sed -n '/<head>/,/<\/head>/p' | grep -c '<title>'      # 1 = title in <head>
curl -s -A "$UA" "$URL" | grep -c 'rel="canonical"'                              # 1
curl -s -A "$UA" "$URL" | sed 's/<[^>]*>/ /g' | wc -w                            # hundreds, not dozens
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/llms.txt          # 200
```

Run `next build` too: its route table shows which routes are static and which render per request, so a page you expected to be static stands out.

Two limits of this test. The user agent only exercises rules that read it, such as `htmlLimitedBots` and some firewall rules; it can’t imitate a crawler that is verified by IP. And a 200 from your laptop doesn’t prove a 200 from a bot: your host or CDN may treat AI crawlers differently, so run the [AI crawler checker](https://katman.pro/tools/ai-crawler-checker) or the curl loop on [Cloudflare blocking AI crawlers](https://katman.pro/cloudflare-blocking-ai-crawlers) as well. For Googlebot, Search Console’s URL inspection shows the HTML it really received.

## Where Katman fits, and where it doesn’t

Katman is an MCP server inside Cursor, Claude Code, VS Code and other coding tools. `katman_scan_code` flags pages whose text loads in the browser; `katman_generate` writes robots.txt, llms.txt, JSON-LD and a Next.js middleware bot counter from one research file, so the files match your pages; `katman_audit_site` checks the live site as AI crawlers fetch it. Your coding tool makes the changes, not Katman. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped ([what the MCP does](https://katman.pro/mcp)). No setup can promise an assistant will recommend you.

Written by the Katman team, Rnify Limited. Next.js docs and repository, OpenAI and Vercel pages read 15 October 2026.

## Who this is not for

-   Next.js Pages Router projects: the file conventions here are App Router ones.
-   Apps where every page sits behind a login: keep them out of search.
-   Anyone after ranking tricks: this page is about making Next.js pages readable and well described.

## Frequently asked questions

### Where do I put llms.txt in a Next.js App Router project?

In public/llms.txt, which Next.js serves at /llms.txt. If you want it generated from your data, write it to public/ in a prebuild script, or use a route handler at app/llms.txt/route.ts and make sure it is cached.

### Do Client Components hurt SEO in Next.js?

Not by themselves. Next.js prerenders Client Components to HTML on the server too. The text goes missing when a Client Component fetches it in useEffect, or when a component is imported with ssr: false.

### Why is my <title> inside <body> when I curl the page?

Streaming metadata. When generateMetadata waits on dynamic data, Next.js sends the page first and appends the tags to the body later, except for user agents on its HTML-limited bots list. GPTBot, OAI-SearchBot and ClaudeBot aren’t on that list.

### Should I generate robots.txt with robots.ts or write a static file?

Either works. app/robots.txt is fine for a fixed file; app/robots.ts is handy when rules differ between production and preview deployments.

### My site was built with v0. Does this apply?

Yes. v0 produces Next.js App Router apps, so the same checks apply; look closely at pages marked use client that load their text in useEffect.

### Does Next.js have its own llms.txt?

For its documentation, yes: nextjs.org serves /docs/llms.txt and Markdown versions of every docs page. That is Vercel’s docs site, not a feature that writes llms.txt for your app.

## Related pages

-   [**Is React bad for SEO, or is it my single-page app?** React is fine for SEO; an empty SPA shell is not. Google renders JavaScript in a queue, the crawlers behind ChatGPT and Claude read only raw HTML. Test, then fix.](https://katman.pro/react-spa-seo-ai-crawlers)
-   [**Do AI crawlers like GPTBot and ClaudeBot render JavaScript?** In Vercel and MERJ’s Dec 2024 log study, GPTBot, ClaudeBot and PerplexityBot didn’t run JavaScript; Google and Apple did. What’s measured, what’s assumed, a test.](https://katman.pro/do-ai-crawlers-render-javascript)
-   [**Is Astro good for SEO and AI search, and what still needs setting up?** Astro ships plain HTML by default, so AI crawlers can read it. Still on you: the sitemap integration, robots.txt, llms.txt, JSON-LD and client:only islands.](https://katman.pro/astro-seo-ai-search)
-   [**What is llms.txt, and is it worth writing?** llms.txt is a short Markdown summary of your site for language models. Google ignores it and most files get no requests; here is when it's still worth writing.](https://katman.pro/what-is-llms-txt)
-   [**llms.txt generator** Enter your domain: we read robots.txt and your sitemap, open up to 30 pages and draft an llms.txt with a title and description per page. Free, no sign-up.](https://katman.pro/llms-txt-generator)
-   [**I built my site with Lovable, Bolt or Cursor, and ChatGPT never mentions it** Built it with Lovable, Bolt, v0 or Cursor and ChatGPT never mentions it? Six traps hide vibe-coded sites; each has a one-minute test and a fix for your AI tool.](https://katman.pro/vibe-coded-site-invisible-to-chatgpt)
-   [**Katman MCP** Katman MCP runs in Cursor, Claude Code and VS Code: it scans your code, audits your live site like AI crawlers do and checks every fix. Katman Pro from $99/month.](https://katman.pro/mcp)

Updated 20 October 2026 · Katman