---
title: "Does the ChatGPT API answer like the app? A study"
description: "Same 3 questions, API panel vs logged-out ChatGPT, 5 Sep 2026: brand presence agreed 3 of 3, competitor lists overlapped 58%, the first brand matched 1 of 3."
url: https://katman.pro/agreement-study
lang: en
updated: 2026-10-16
---

Case study Updated 16 Oct 2026

# Does the API panel say what ChatGPT says?

> On 5 September 2026 we asked three buyer questions in our own category twice: through the API, with the exact call our visibility engine made, and in logged-out chatgpt.com the same day. Whether the target brand appeared agreed in 3 of 3 pairs. The other brands named overlapped 58% on average, the brand named first matched in 1 of 3, and the app skipped search on one question. Three pairs: a dated observation, not a rule.

## About this study

Every AI visibility product measures through the providers’ APIs, ours included. A buyer sits in front of a consumer app with its own system prompt, its own decision about whether to search, and possibly memory of them. We had always said the two may differ. This study measures by how much, on a small dated sample.

-   **When:** 5 September 2026, one pass, both sides the same day (own measurement).
-   **API side (“the panel”):** the exact request our visibility engine made at the time. OpenAI’s `gpt-5.6-sol` through OpenRouter, the same system prompt, temperature 0.2 and the provider’s own (native) web retrieval.
-   **App side:** chatgpt.com, logged out, no account, default model, web search on by default.
-   **Category:** our own, AI visibility measurement. Target brand: Katman. Three questions, so three pairs.

## The result in one table

| Question a buyer would notice | Agreement | What it means |
| --- | --- | --- |
| Does the target brand appear at all? | 3 of 3 | Katman was absent on both sides every time. The core measurement, “is this brand in the answer”, matched. |
| Which competitors are named? | 58% | Average overlap of the brands named. The lists share a core and differ at the edges. |
| Which brand is named first? | 1 of 3 | Position inside an answer was the least stable thing we measured. |
| Did the app search at all? | 2 of 3 | For “llms.txt generator tool” ChatGPT answered from memory and named nobody. |
| Which domains are cited? | 4% | Low confidence: logged-out ChatGPT showed sources as labelled chips, not links. |

## Every question, both sides

| Question (verbatim) | API named | ChatGPT app named | Named first (API / app) | App searched | Overlap |
| --- | --- | --- | --- | --- | --- |
| “Best AEO (answer engine optimization) tools 2026” | Profound, Peec, Scrunch, Semrush, Ahrefs, Writesonic | Profound, Peec, Scrunch, AthenaHQ, Otterly, Semrush, Ahrefs, Writesonic | Profound / Profound | yes | 75% |
| “What tools can audit whether AI chatbots like ChatGPT mention or cite my website?” | Profound, Peec, Scrunch, Otterly, Semrush, Ahrefs | the same six | Otterly / Profound | yes | 100% |
| “llms.txt generator tool” | Trakkr, Mintlify, AIOSEO | none | AIOSEO / none | no | 0% |

The second question is the clearest case: both sides named the same six products, and still put a different one first.

### Example: the llms.txt pair

The app’s whole answer was an offer: it could generate an llms.txt file if you sent your URL. No search, no product names. The API answer, with retrieval attached, recommended a browser generator, a WordPress plugin, an open-source generator and Mintlify’s built-in option, with sources. Same question, same day, two different experiments: one answered from memory, one from fresh search results. If you only measure through the API, you never see the first kind.

## What this means for reading visibility numbers

1.  **Presence holds.** Named, recommended and cited for the target brand are the numbers a report leads with, and on this sample they matched the app every time.
2.  **Competitor lists are indicative.** On 5 September 2026 the panel’s competitor set agreed 58% with logged-out ChatGPT. Use it to see who keeps coming up, not as a census.
3.  **Position is noise** until a larger sample says otherwise. One pair in three agreed on the brand named first.
4.  **“Did not search” is a result.** An answer from memory is a different experiment from a retrieved one, and a report should record it per question instead of hiding it.

The same caution comes from larger outside studies; the [AI visibility tools comparison](https://katman.pro/ai-visibility-tools) cites one. To check your own brand by hand, the steps are on [Check if ChatGPT recommends your site](https://katman.pro/check-if-chatgpt-recommends-your-site).

## Method, and what it can’t claim

-   **Three pairs.** Enough to publish, not enough to generalize.
-   **One app.** Claude.ai and Gemini needed an account, and a signed-in account carries memory and personalization, the confound we wanted to keep out. Six API answers from Anthropic and Google in this category were recorded and have no app pair.
-   **A fixed brand list.** Brands were counted against a lexicon of 46 names built from both sides’ answers before comparing. A brand not on it is invisible to every rate above: the API’s llms.txt answer led with a generator that wasn’t on the list. Names that are also ordinary words count only in unambiguous forms, so the app’s mentions of Conductor and Surfer in the first answer were not counted.
-   **Cited domains are low confidence.** The app’s source chips were mapped to domains only where the label was unambiguous.
-   **One pass, one day.** Temperature 0.2 on the API side, the app’s default on the other. A single run is noise, here as anywhere.
-   **Cost.** $1.78 for the day’s 18 API answers with native retrieval; nine are on this page, the other nine belong to a second category we don’t publish. OpenAI calls that timed out or came back empty were retried.

Raw data, as published on 5 September 2026: [API answers](https://katman.pro/agreement/api-2026-09-05.json), [app answers](https://katman.pro/agreement/consumer-2026-09-05.json), [computed comparison](https://katman.pro/agreement/report-2026-09-05.json), [brand lexicon](https://katman.pro/agreement/lexicon.json), [questions](https://katman.pro/agreement/queries.json).

## What changed since 5 September 2026

The product that ran this study was a hosted audit service. Katman is now [an MCP server](https://katman.pro/mcp) that runs inside your AI coding tool (Cursor, Claude Code, VS Code, Windsurf, Codex CLI, Gemini CLI or Claude Desktop). Three things from this study are built into it:

-   `katman_visibility_run` records named, recommended first and cited separately, and whether the model actually searched, so “did not search” shows up as a state. It asks ChatGPT, Claude, Gemini, Perplexity and Grok through OpenRouter on your own key, about $2.40 a run on the default panel. The models are newer than the ones above, so these numbers don’t transfer to it.
-   `katman_manual_check` records an answer you paste from the app yourself, logged out, so you can build your own pairs for the questions that matter.
-   The [visibility runs docs](https://katman.pro/docs/visibility-runs) carry the same reading rule: presence fairly reliable, rank unreliable.

Katman doesn’t edit your site by itself; your coding tool makes the changes. Katman Pro is $99 a month for up to 3 sites, with fix prompts and page briefs uncapped. Or have us do it: [Growth](https://katman.pro/done-for-you) writes 30 pages a month for you. Nobody can promise an assistant will recommend you, and this study is one reason why: even the measurement moves.

## Who this is not for

-   Choosing an AI visibility tool: this page measures how far API answers match the app, not which tool to buy. For that, see the tools comparison.
-   Conclusions about the Claude, Gemini, Perplexity or Grok apps: only logged-out ChatGPT was paired.
-   A forecast for your brand or category: three questions, one category, one day.

## Frequently asked questions

### Do API answers match what people see in the ChatGPT app?

Partly. In our 3 pairs on 5 September 2026, whether a brand appeared matched every time, the list of other brands overlapped 58% on average, and the brand named first matched once. Three pairs is a small sample, so read it as a direction, not a rate.

### Why did you only pair ChatGPT, not Claude or Gemini?

Their consumer apps needed an account, and a signed-in account brings memory and personalization, which is exactly what we wanted to keep out. Their API answers were recorded; they just have no app answer to compare with.

### What does “58% overlap” mean here?

For each question we took the brands both sides named and divided by all brands either side named (Jaccard overlap), then averaged the three: 75%, 100% and 0%.

### Why did the llms.txt question score 0%?

Logged-out ChatGPT answered from memory, did not search and named no product. The API call always had web retrieval attached, so it named three. An API panel can’t see an app’s decision not to search.

### How should I read my own visibility numbers after this?

Treat “is my brand in the answer” as the solid number, the list of competitors as indicative, and position as noise until repeated runs agree. Check the two or three questions that matter most by hand in the app.

### Has the study been repeated since?

Not with this harness. This page reports the single pass of 5 September 2026; a rerun would be published with its own date next to it, not instead of it.

## Related pages

-   [**Do LLMs know we exist? A six-model visibility test of katman.pro** 6 models, closed-book and with web search, Sep 2026: Katman was named in 0 of 25 valid discovery answers and 0 of 84 problem-query answers. Raw counts and limits.](https://katman.pro/llm-visibility-report)
-   [**Who do AI assistants name for AI visibility tools, and which pages do they read?** 20 answers from 5 AI families on 5 Sep 2026: Profound was named in 19, Peec AI in 18, Katman in 0. The 12 brands named and the pages the models read to decide.](https://katman.pro/who-ai-names-for-ai-visibility)
-   [**How do I check whether ChatGPT, Claude and Perplexity recommend my site?** Ask 12 unbranded questions and 2 “What is \[brand\]?” ones logged out, then log who is named, recommended and cited. Why one answer misleads, and the API option.](https://katman.pro/check-if-chatgpt-recommends-your-site)
-   [**Visibility runs** How Katman measures AI visibility: 12+2 question set, five assistants via OpenRouter, cost caps, manual checks in the apps, and why one run is a small sample.](https://katman.pro/docs/visibility-runs)
-   [**Which AI visibility tool fits which situation?** Profound, Peec AI, Otterly.AI, Semrush, Ahrefs and seven more AI visibility tools: what each does, its price on 30 Sep 2026, who it fits and its main weakness.](https://katman.pro/ai-visibility-tools)

Updated 16 October 2026 · Katman