How do I set up Next.js for AI search, and where does llms.txt go?

Next.js already sends HTML; here is what undoes it

App Router pages are Server Components by default, and Next.js uses Client Components and the server payload “to prerender HTML” as well (Next.js, “Server and Client Components”, updated 5 Oct 2026, read 15 Oct 2026). A Next.js page is readable without JavaScript unless something takes the text out of the HTML:

• Data fetched in the browser. A "use client" page that loads its copy in useEffect ships the loading state. Fetch in the Server Component and pass the data down as props.
• next/dynamic with ssr: false. Whatever that component draws is missing from the HTML.
• Streaming metadata. Since 15.2, when generateMetadata waits on dynamic data, Next.js sends the page first and appends the tags “to the <body> tag” later. Bots on its HTML-limited list get them in <head> instead (generateMetadata, updated 19 Aug 2026). The default list (in the Next.js repository, read 15 Oct 2026) includes Bingbot, applebot and Google’s special crawlers, but not GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot.

For that last one, keep metadata static or cached where you can. If it must be dynamic, extend the list. Your regex replaces the default one, so keep the bots you still need:

// next.config.ts
import type { NextConfig } from "next";

const config: NextConfig = {
  htmlLimitedBots:
    /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Bingbot|applebot|facebookexternalhit|Twitterbot|LinkedInBot|Slackbot/i,
};
export default config;

v0 produces Next.js App Router apps, so a v0 site follows the same rules (our builder table).

Metadata, robots.ts and sitemap.ts

Set the base URL and a title template once, then give every page its own title, description and canonical:

// app/layout.tsx
export const metadata: Metadata = {
  metadataBase: new URL("https://yoursite.com"),
  title: { template: "%s | Yoursite", default: "Yoursite" },
};

// app/pricing/page.tsx
export const metadata: Metadata = {
  title: "Pricing",
  description: "Plans, limits and what each one includes.",
  alternates: { canonical: "/pricing" },
};

Robots rules per AI bot, with the app area closed:

// app/robots.ts
import type { MetadataRoute } from "next";

const aiBots = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot", "Claude-User", "ClaudeBot", "PerplexityBot", "Perplexity-User"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/", disallow: ["/app/", "/api/"] },
      { userAgent: aiBots, allow: "/", disallow: ["/app/", "/api/"] },
    ],
    sitemap: "https://yoursite.com/sitemap.xml",
  };
}

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getPages } from "@/lib/pages";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const pages = await getPages();
  return pages.map((p) => ({ url: `https://yoursite.com${p.path}`, lastModified: p.updatedAt }));
}

Both files are route handlers that are cached by default (robots, updated 1 May 2026; sitemap, updated 18 Aug 2026). Use lastModified for the date the words changed, not the build date.

Where llms.txt goes

Next.js has no llms.txt file convention. Two options:

The static file is the simpler of the two. Generate it before each build from the same data your pages use:

// scripts/llms.mjs   (package.json: "prebuild": "node scripts/llms.mjs")
import { writeFileSync } from "node:fs";
import site from "../content/site.json" with { type: "json" };

const lines = [`# ${site.name}`, "", `> ${site.oneSentence}`, "", "## Pages", "",
  ...site.pages.map((p) => `- [${p.title}](https://yoursite.com${p.path}): ${p.summary}`)];
writeFileSync("public/llms.txt", lines.join("\n") + "\n");

Keep expectations low: Google ignores llms.txt, and most files get no requests at all (What is llms.txt?). It is cheap, and useful only when it says exactly what your pages say. A starting file comes out of the llms.txt generator.

JSON-LD

Next.js recommends a plain <script type="application/ld+json"> in the page, with JSON.stringify(jsonLd).replace(/</g, "\\u003c") to escape < (Next.js, JSON-LD guide, updated 2 Mar 2026). Put the same facts in it that the visible page states.

Check it as a bot (example)

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'
URL=https://yoursite.com/pricing
curl -s -A "$UA" "$URL" | sed -n '/<head>/,/<\/head>/p' | grep -c '<title>'      # 1 = title in <head>
curl -s -A "$UA" "$URL" | grep -c 'rel="canonical"'                              # 1
curl -s -A "$UA" "$URL" | sed 's/<[^>]*>/ /g' | wc -w                            # hundreds, not dozens
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/llms.txt          # 200

Run next build too: its route table shows which routes are static and which render per request, so a page you expected to be static stands out.

Two limits of this test. The user agent only exercises rules that read it, such as htmlLimitedBots and some firewall rules; it can’t imitate a crawler that is verified by IP. And a 200 from your laptop doesn’t prove a 200 from a bot: your host or CDN may treat AI crawlers differently, so run the AI crawler checker or the curl loop on Cloudflare blocking AI crawlers as well. For Googlebot, Search Console’s URL inspection shows the HTML it really received.

Where Katman fits, and where it doesn’t

Katman is an MCP server inside Cursor, Claude Code, VS Code and other coding tools. katman_scan_code flags pages whose text loads in the browser; katman_generate writes robots.txt, llms.txt, JSON-LD and a Next.js middleware bot counter from one research file, so the files match your pages; katman_audit_site checks the live site as AI crawlers fetch it. Your coding tool makes the changes, not Katman. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped (what the MCP does). No setup can promise an assistant will recommend you.

Written by the Katman team, Rnify Limited. Next.js docs and repository, OpenAI and Vercel pages read 15 October 2026.