How confident should you be in an AI visibility number? The statistics behind the percentage

Why a bare percentage is the least trustworthy number on a dashboard

"We are mentioned in 40% of answers" sounds precise. It is not, unless you also know how many answers that 40% came from. Two of five is 40%. Forty of one hundred is also 40%. A reasonable person trusts the second one far more than the first, and any dashboard that shows both as the identical number "40%" is quietly lying by omission. This is the single most common failure mode in AI visibility reporting: a tool runs a handful of prompts, computes a clean-looking percentage, and presents it with the same confidence as a number backed by hundreds of observations.

The fix is not complicated, but it has to be applied consistently, everywhere a rate appears, or it does not work. Three rules do the job: never show a percentage below a minimum sample, always compute and show an interval around the ones you do show, and treat repeated runs of the same question as one piece of evidence, not several.

Rule one: below five completed runs, show a count, not a percentage

A rate computed from fewer than five completed, eligible runs is shown as a plain count, such as '2 of 4 runs', with no percentage and no gauge. This is a deliberate floor, not a rounding choice. A percentage implies a rate has been measured precisely enough to compare against a threshold or a competitor. At three or four observations it has not been, and showing 67% instead of '2 of 3' invites exactly the false precision the whole system exists to avoid.

Rule two: the Wilson score interval, widened by a prompt-cluster bootstrap

Once there are enough runs for a percentage, the number still needs a stated margin of uncertainty. The classic approach, a normal approximation interval, behaves badly exactly where AI visibility data lives: small samples and rates near 0% or 100%. It can produce a lower bound below zero or an upper bound above one hundred, which is obviously wrong and which anyone who has used a basic online confidence-interval calculator has probably seen happen.

The Wilson score interval does not have this problem. It is a standard method in the statistical literature for binomial proportions (the technical name for a yes/no rate like 'was the brand named'), and it stays inside the 0 to 100% range and behaves sensibly at small sample sizes and at the extremes, which is precisely the regime AI visibility rates usually sit in: dozens of runs, not thousands, and rates that are often genuinely low or genuinely high rather than near 50%.

That interval alone is not enough, because it assumes every run is an independent coin flip, and that assumption is false the moment the same prompt is asked more than once. If a brand is asked about in the same way five times and the answers are five restatements of the same underlying fact from the model, that is not five independent data points, it is one data point repeated five times with some noise on top. Treating it as five independent flips makes the interval look tighter than the evidence actually supports.

The correction is a cluster bootstrap: instead of resampling individual runs, whole prompts are resampled, with every run that belongs to a resampled prompt coming along with it. This produces a second interval that respects the real unit of independent evidence, which is the prompt, not the run. The final interval shown on screen is the wider of the two, the Wilson interval and the bootstrap interval, so correlation between repeats can only ever make the stated uncertainty more honest, never less.

Rule three: repeats agreeing or disagreeing is its own number, not hidden inside the average

When the same prompt is asked of the same engine more than once in one audit, two further things become measurable that an average alone hides: how often the repeats agree with each other, and how often they flip from naming the brand to not naming it. A prompt where three repeats all name the brand is a different, more reliable kind of evidence than a prompt where two name it and one does not, even if both pairs average out to the same overall visibility rate across the whole audit.

Agreement is reported as the mean share of repeats that side with whichever answer (named or not named) was in the majority for that prompt and engine. The flip rate is the share of repeated prompt-and-engine pairs where the repeats did not all agree. A high flip rate on a specific prompt is a useful, actionable signal on its own: it usually means the engine is genuinely uncertain for that question, which is different from, and more informative than, a single overall percentage that quietly averages the disagreement away.

Stability across audits: the same rate, over time, held to the same standard

A single audit's interval answers 'how much should I trust this one number'. A separate question is whether a rate has genuinely held steady across several audits over time, or whether it has swung around more than normal audit-to-audit noise would explain. That needs more than two points. The method here waits for at least three audits that each individually clear the five-run floor, computes the mean and the range (highest minus lowest) across those audits, and derives a coefficient of variation: the standard deviation of the rate across audits divided by its mean.

Below a set threshold on that coefficient, the rate is labelled stable; above it, variable. Fewer than three qualifying audits, and the honest answer is that there is not yet enough history to call it either, so it is labelled exactly that: not enough data, rather than quietly assumed to be stable by default.

What this method deliberately does not do

• It does not pretend a wider interval is a failure. A wide interval on a small sample is the correct, honest description of a small sample.
• It does not average away disagreement between repeats into a single smoothed number without also showing the disagreement itself.
• It does not call a rate stable from two data points, no matter how similar those two points happen to look.
• It does not inflate confidence by quietly treating repeated runs of one prompt as if each repeat were a fresh, independent observation.