One AI screenshot is not a measurement

Someone on your team asks an AI assistant about your category, your brand does not appear, and a screenshot lands in the group chat. It feels like evidence. It is closer to a single coin toss.

AI answers are sampled, not looked up

A search engine returns a ranked list that is broadly stable for a given query. An AI assistant writes a fresh answer each time. The same question can produce different brands, in a different order, from one run to the next, and differently again across model versions, locations and accounts.

That means one answer is one draw from a range of possible answers. It can be typical or it can be an outlier, and you cannot tell which from a single example.

What to do instead

1. 1Define a set of prompts that represent how real customers ask, grouped by topic and intent.
2. 2Run each prompt many times across every platform you care about.
3. 3Report rates with the sample size shown, for example mentioned in 12 of 40 eligible answers.
4. 4Keep the full answers so any figure can be traced back to what was actually said.

Why the denominator matters

A mention rate is only meaningful against a defined set of eligible observations. Dividing by an arbitrary total makes results look better or worse than they are. BrandWater uses eligible observations only and shows the count beside every rate.

None of this makes a screenshot useless. It is a good prompt for a question worth measuring. It is just not the measurement.

Why the same question gives different answers

Four things move an AI answer from one run to the next. The first is sampling: a language model chooses each word from a range of likely words, so two runs of the same prompt are rarely identical. The second is retrieval: when web search is switched on, the pages fetched can differ by moment, location and index freshness. The third is the model itself, because providers update models and search behaviour without announcing every change. The fourth is context: the account, the conversation history, the country and the language of the person asking.

None of these is a defect. They are how the systems work. The consequence for a brand team is that any single answer belongs to a distribution. It can sit in the middle of that distribution or far out in a tail, and one observation cannot tell you which.

What a real observation needs to record

A screenshot captures the words on a screen. A measurement captures the conditions that produced them, so that someone else can repeat the run and compare. A useful observation record has these fields:

• The exact prompt, word for word, and the topic and intent it belongs to.
• The platform and, where visible, the model or mode used (with or without web search).
• The date and time, the country and the language of the request.
• The full answer text, not an excerpt.
• Every source link shown with the answer, in order.
• Whether the run completed, so failed runs are excluded from the denominator rather than counted as misses.

How sample size changes what you can conclude

The uncertainty around a rate shrinks with the square root of the number of runs. In practice that means quadrupling the number of observations roughly halves the margin of error. A mention rate built on a handful of runs can move a long way when you repeat it, while one built on many runs settles. This is why a reported rate should always sit next to its sample size, and why a team should not react to a change that is smaller than the noise in the measurement.

It also explains a common trap. If a report shows a brand rising from one week to the next, the first question is whether the same prompts were used on the same platforms with a similar number of runs. Without that, an increase is as likely to be a change in method as a change in reality.

Where a screenshot is still useful

A screenshot is a good starting point. It is evidence that a question is worth measuring, a way to show a colleague what a customer might see, and a record of one specific moment. Teams often use them well as illustrations inside a report, as long as the report also states the rate they illustrate and how many runs sit behind it.

A four week starting routine

1. 1Week one: write twenty to thirty prompts that mirror real customer questions and group them by topic and intent.
2. 2Week two: run every prompt several times on each platform you care about and store the full answers with their sources.
3. 3Week three: calculate mention, recommendation and citation rates against eligible answers and record the sample size beside each.
4. 4Week four: read the answers behind the lowest rates, list the gaps you find, and repeat the same run to see how stable the numbers are.