100 ChatGPT Reruns Cut Site Visibility Measurement Error to ±10 Points

Yesterday ChatGPT cited your site; today the same question returns no link. That does not yet mean the site has lost rankings. Answers from AI search change from run to run: the model rephrases the original question, sends several internal search queries, retrieves different pages and builds an answer from them.

That is why a single run gives only a "yes" or a "no". It does not show how often the site appears in answers. For an analytics developer, a product team or a business owner, a different metric is more useful: the share of reruns in which the domain was among the sources, together with a range of statistical uncertainty.

One answer measures nothing

Suppose a site actually appears in 30% of answers. In any single run, the probability of not seeing it is 70%. The link could be there yesterday and vanish today without any changes to the site, the model or the search index.

To get a number, the same question must be asked many times under identical conditions. If the domain appeared in 6 answers out of 20, the observed visibility is 30%. But the sample is small: the 95% Wilson interval would be roughly 15% to 52%. In other words, the result is still consistent both with the site appearing rarely and with it appearing in almost every second answer.

With 30 hits out of 100, the share stays the same, 30%, but the interval narrows to about 22–40%. Such a measurement is already noticeably more useful for comparing periods and testing hypotheses.

The Wilson interval is more convenient here than the simple formula "share plus or minus standard error". With small samples and rare hits, the simple formula can produce a meaningless negative lower bound. The Wilson interval stays within the 0% to 100% range and behaves better with few observations.

How many runs you need

The number of reruns depends on the acceptable error. At 95% confidence, a conservative estimate gives about 25 runs for an error of ±20 percentage points, 43 for ±15 points, 97 for ±10 points and 385 for ±5 points. The calculation assumes a 50% share, which requires the largest sample; for a site that appears rarely, the actual interval may be narrower.

A practical compromise is 60–100 reruns for each important question. One hundred runs give an error of about ±10 points in the worst case. If there are fewer than 25 runs, the report is suitable more for exploration than for a confident conclusion about growth or decline.

Before launching, you need to fix verbatim the question, region, memory state, model and hit criterion. Decide separately whether you count only a link visible to the user or any page the system viewed during search. The criterion must not be changed between measurements.

The API and the chat show different things

A measurement through the API does not necessarily reflect what a person sees in the ChatGPT interface. The API may return a full list of pages viewed, whereas the chat shows only some of the citations. In addition, the model sometimes decides not to use search at all. Recording such an answer as a miss is acceptable, but the rule must be the same across all series.

Therefore, the report must state the measurement channel. A share obtained from the API's internal sources cannot be called user-facing visibility in ChatGPT without caveats.

Comparing two periods only by whether individual intervals overlap is also insufficient: it is a convenient rough check, but not a full test of the difference between proportions. For an important decision, it is better to keep the raw results and apply a statistical test that accounts for two samples.

The practical minimum for a report is simple: the exact question, run conditions, number of reruns, number of hits, share and 95% interval. If a contractor shows one successful answer or a single final figure without the sample size, that is a demonstration, not a measurement.