Do LLMs know we exist? A six-model visibility test of katman.pro

About this test

We put our own measurement protocol on ourselves and published the raw counts. Two runs, both our own measurement through OpenRouter:

• 3 September 2026, the question matrix. Six models, six questions, each asked once closed-book (training data only) and once with web access: 72 calls.
• 2 September 2026, the problem-based test. The engine read our own pages, listed the problems Katman solved, wrote the queries a buyer would type without the brand name, and asked the same six models each one, closed-book and with web access: 108 calls.

The six models: OpenAI gpt-5.6-sol, Anthropic claude-sonnet-5, Google gemini-3.6-flash, Meta llama-3.3-70b-instruct, DeepSeek deepseek-chat-v3.1 and Mistral mistral-small-3.2-24b-instruct. One request per cell, temperature 0, at most 500 tokens per answer. A “mention” is an answer that contains “katman” or “katman.pro”.

The result in one table

The 3 September run was marked inconclusive: with 29 of 72 answers valid, the zero for discovery stands, but no model can be compared with another.

How to read it: discovery vs branded

Only discovery rows measure visibility. Five of the six questions on 3 September were discovery questions:

1. “What tools can audit whether AI chatbots like ChatGPT mention or cite my website?”
2. “Best AEO (answer engine optimization) tools 2026”
3. “How can I get my SaaS recommended by ChatGPT when users ask for alternatives?”
4. “llms.txt generator tool”
5. “How to check if Cloudflare is blocking GPTBot on my site”

The sixth was simply “katman.pro”, a navigational question. It types the domain, so an answer contains it almost whatever the model knows. That is what the 4 of 4 shows. In the recorded excerpts, three web answers said the search results had nothing direct about katman.pro, and the closed-book answer guessed it was a proxy or VPN service. Branded recall of 4 of 4 is a reminder of why branded questions are reported separately and never as discovery.

The 3 September matrix, model by model

Claude and Gemini returned no valid answer at all, so this run says nothing about them. The questions about getting a SaaS recommended, llms.txt generators and Cloudflare named none of the tracked vendors in any valid answer.

The problem-based test, 2 September 2026

The engine diagnosed five problems from our pages (blocked AI crawlers, llms.txt, no view of LLM citations, technical issues, comparing closed-book and web answers) and wrote nine queries. Each was asked 12 times: six models, closed-book and web.

Totals: 108 calls, 24 failed (21× HTTP 402, 2× HTTP 429, 1 timeout), 84 valid answers, 0 naming Katman. Across the valid answers, Profound was named in 8, Otterly.ai in 5, Peec AI in 4, Screaming Frog and Scrunch AI in 1 each.

Example: one recorded row from the problem test, as stored in the run file (excerpt cut).

{
  "q": "best tools for Generative Engine Optimization GEO",
  "model": "openai/gpt-5.6-sol",
  "mode": "closed",
  "brandMentioned": false,
  "competitorsFound": ["Profound", "Otterly.ai", "Peec AI", "Scrunch AI"],
  "error": null
}

Limits

• Small samples. 25 and 84 valid answers, one pass each. Zero is a clear result at this size; the shares of the other tools are not.
• English only, APIs only. A brand can be absent in one language and named in another, and an API answer is not the surface a person sees in the app. How far the two matched on 5 September is in the agreement study.
• A short vendor list. Tool names were matched against a list of known AI visibility vendors. The web answer from gpt-5.6-sol to the llms.txt question pointed to Writesonic’s generator, which wasn’t on that list.
• 500 tokens. Long answers were cut, so a name late in an answer could be missed.
• Failures are not absences. Every failed call is counted as a failure, never as “not named”.

What changed since September 2026

When this ran, Katman was a hosted audit service. It is now an MCP server that runs inside your AI coding tool, and the same measurement is one tool in it. katman_visibility_run asks a fixed set of 12 unbranded questions plus 2 diagnostics to ChatGPT, Claude, Gemini, Perplexity and Grok through OpenRouter on your own key, about $2.40 a run on the default panel, with a dry run and a cost cap first. It reports named, recommended and cited separately and keeps failed calls apart from absences, as this page does (visibility runs).

What Katman adds beyond the measurement is the fix work: it reads your code and live pages the way AI crawlers do, then tells your coding tool what to change and which pages to write. Katman doesn’t edit your site by itself. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped. Or have us do it: Growth writes 30 pages a month for you. Nobody can promise an assistant will recommend you; for the usual reasons it doesn’t, see Why doesn’t ChatGPT recommend my brand?.