Why does one answer fail as evidence?
A single good answer tells you it happened once, not that it happens. OpenAI's own documentation on ChatGPT search explains that ranking is based on a number of factors and there is "no way to guarantee top placement". It also says ChatGPT search "typically rewrites your query into one or more targeted queries" before it goes looking for results. That matters: the phrasing you typed is not the phrasing the system actually ran. Test one clever prompt and you have measured that prompt's luck, not your visibility. A prompt set spreads the risk across enough variation that a pattern can show through.
Google describes something structurally similar for its AI features: a "query fan-out" technique that lets these systems surface a wider and more diverse set of links than a single query would. Different platforms, same implication. The input you control is a starting point the system reworks, not the final query.
How do you build the prompt set?
Derive prompts from real buyer language, not from your own product vocabulary. Pull phrasing from sales calls, support tickets, review sites, and search terms you already know buyers use. Then cover the stages of a real buying journey rather than repeating one angle:
- Category discovery. "what are the best [service] providers for [need]."
- Comparison. "[option A] vs [option B] for [use case]."
- Best-for-Y framing. "best [service] for [specific outcome or constraint]."
- Local or qualified variants, adding a location, budget, or industry that narrows the field.
Keep the set small enough that you will actually repeat it. Ten to fifteen prompts is a reasonable range. Then freeze it. Changing the wording between rounds breaks the comparison; you would no longer know whether an answer moved because the market moved or because you asked a different question.
Once a round has run, the prompt set is locked. New prompts start a new set, not a revision of the old one.
What controls keep the test honest?
Run every prompt from a temporary or logged-out session. OpenAI's memory documentation explains that ChatGPT can remember context from past chats to personalise responses, through saved memories and chat history, and that this can be turned off in Settings. Its Temporary Chat documentation confirms that Temporary Chats "do not use existing memories or create new memories" and will not appear in history. If you test while logged into an account that has spent months researching your own brand, you are measuring your own history, not the market's view. Temporary Chat, or a logged-out session, is the control condition.
Beyond that: note the model and the date for every run, since both are moving targets. And run each prompt more than once per round, because outputs vary between runs on the same system, so a single pass through the list can mislead you either way.
It can tell you, for a fixed set of questions, whether your brand shows up, whether it is cited, whether the description is accurate, and whether that changes over time. It cannot tell you why an individual answer came out a particular way, cannot guarantee the next unscripted question behaves the same, and cannot prove causation between a change you shipped and a change in the answers is only correlation worth investigating further.
What should you log each round?
Track these as separate columns, because they are different achievements. Collapsing them into one score hides which problem you actually have.
Was the brand named at all, anywhere in the answer?
Was the site linked or cited as a source?
Was what it said about the business correct?
Did a competitor get named in the same slot instead?
A brand that is mentioned but never cited has a different problem to one that is cited but described incorrectly. Keep the columns apart so the next action is obvious.
Draft the prompt set from real buyer language and freeze it.
Run the full set under controlled conditions and log every column.
Repeat on a fixed schedule after changes ship, not daily.
Keep, revise, or stop based on the logged pattern, not one answer.
Re-run on a fixed cadence tied to real changes, such as a site update, a new page, or a PR push, rather than checking every day. Daily checks mostly surface the variation between runs, and you end up reacting to noise.
How do you handle a run that fails?
A run that errors is not a run where the brand was absent. Pooling those two corrupts every rate you calculate.
Mark each run valid or invalid before reading it, and record the reason. Common invalid cases: the surface returned an error or refused, the search mode did not engage, the session carried context from a previous prompt, the platform was unavailable in that region, or the answer addressed a different question entirely.
- An invalid run leaves the denominator. It is logged, not counted.
- A valid run with no brand mention stays in. That is a real negative and the most common useful result.
- Report the valid-run count with every rate. "3 of 12 valid runs" is interpretable. "25%" on its own is not.
If more than a small share of a wave is invalid, fix the collection before comparing waves. A drop caused by a platform outage looks identical to a drop caused by losing visibility.
Record which URL was cited, not just which domain
"We were cited" hides the distinction that matters most. Log them separately:
| Observation | What it means |
|---|---|
| Exact page citation | The specific URL you registered was displayed. The strongest observation available |
| Same-domain citation | Your domain appeared, but a different page. Useful, and not evidence that the page you were testing worked |
| Syndicated copy | A republished version of your content on another domain. Give it its own URL and a shared story identifier |
| Third-party source | Someone else's page carried the claim. This can be the better outcome and is often the more credible one |
Also separate accuracy from completeness. Everything an answer says about you can be correct while you are absent from a list that should include you. Those are different failures with different fixes, and a single visibility score cannot distinguish them.
Choosing the prompts
The set is the instrument. A prompt list assembled from what you wish people asked measures nothing. For each candidate prompt, record the customer question that prompted it, the buyer stage, the target country, what a genuinely helpful answer would contain, whether your brand appears in the prompt, and what result would change a decision.
Cover the stages rather than repeating one angle. Using a packaging studio as an example:
| Stage | Example | What it measures |
|---|---|---|
| Discovery | "Which independent packaging studios work with food brands in Australia?" | Unaided inclusion in a shortlist |
| Fit | "What evidence should I ask a packaging studio for before hiring them?" | Whether you are cited as a source on your own subject |
| Comparison | "When should a product business hire a branding studio rather than a packaging production specialist?" | Category placement and how you are described |
| Validation | "What packaging work has [studio] published?" | Accuracy, once someone already knows the name |
Only the last one names the brand, and it measures something different. A good validation result with no discovery result means people who already know you get correct information and nobody else finds you. That is a specific, fixable problem, and it is invisible if brand-name prompts dominate your set.
Include a question only if real buyer language supports it: enquiry forms, sales calls, reviews, support tickets. Nobody publishes reliable prompt volume, so a set built from keyword tools is a set built from a different channel's behaviour.
What are the honest limits of this method?
This is observational, not experimental. Sample sizes are small by design, because the point is repeatability, not statistical power. You cannot attribute a change in the answers to any one action you took, because too many other variables move at the same time, including the platform itself. Answers vary between runs of the same prompt; that variation is a known feature of how these systems sample responses, not a flaw in your method, and it is exactly why a fixed set beats one-off checks. And platforms change their retrieval and ranking behaviour without notice, so a prompt set that behaves one way this quarter is not guaranteed to behave the same way next quarter. Treat every result as a directional signal, not a verdict.
A prompt set does not prove why an answer changed. It proves whether it changed, across enough questions and enough runs, to trust the pattern.
OpenAI Help Center: ChatGPT Search
OpenAI Help Center: Memory FAQ
OpenAI Help Center: Temporary Chat FAQ
Google Search Central: AI features and your website
Need the faster version first? Run the 60-second check to find a question worth tracking, then build the frozen prompt set around it.
Run-to-run variance is also why the vocabulary around AI search keeps shifting. Without stable measurement, no definition can be tested.
Where a platform reports its own numbers, use them alongside your prompt set rather than instead of it. Search Console now reports generative AI impressions, but not the prompts behind them, which is exactly the gap a fixed prompt set fills. We took that approach when testing whether Bing indexing affects ChatGPT search visibility.