Explainer

How Accurate Are AI Visibility Tracking Tools? The Numbers

AI visibility tracking tools are accurate at counting what they collect, but each score is an estimate from a sample of answers that vary between runs. At 20% observed visibility, the 95% range is roughly 6% to 51% from 10 answers and 16% to 25% from 300. Always ask for the sample size and interval.

Key points

  • A visibility score is a sample proportion, so it needs a sample size and an interval before it means much.
  • By our Wilson calculation, 20% observed from 10 answers could be anywhere from about 6% to 51%, and from 300 answers about 16% to 25%.
  • OpenAI and Anthropic both document that identical requests can return different outputs, so single runs prove little.
  • How closely API-collected answers match what signed-in users see is not publicly established.
  • Treat a change as real only when a two-proportion check clears chance, then confirm with GA4 referral data such as utm_source=chatgpt.com.

How accurate are AI visibility tracking tools?

AI visibility tracking tools are accurate about what they collected and uncertain about everything beyond it. A tracker can count exactly how many stored answers name your brand. The percentage it shows is still an estimate, because the assistant would give different answers if asked again.

So accuracy has two parts. The first is sampling error: how far the score could sit from the true rate because of the number of answers collected. You can calculate that. The second is coverage: whether the answers a tool collects resemble what your customers see. Nobody can fully calculate that, and vendors rarely publish enough to judge it.

This guide deals with both. It does not repeat how to count mentions against competitors. For that, see the AI share of voice formula.

Why do AI answers differ between runs?

AI answers differ because the models pick words by probability, and the vendors say so in their own documentation. Search-backed assistants add a second source of variation, since the pages they retrieve can change too.

Three vendor documents, all fetched on 11 November 2026, describe this:

  • OpenAI. Its API guide on reproducible outputs says Chat Completions are non-deterministic by default. A fixed seed gives what it calls "(mostly) deterministic" results, and back-end changes can still alter outputs.
  • Anthropic. Its glossary entry on temperature says results are not fully deterministic even at temperature 0. Identical inputs can produce different outputs.
  • Google. Its Gemini prompting guide says temperature controls randomness in token selection. It also says its newest models ignore custom temperature values, so a tracker cannot simply switch the randomness off.

Google's Search Central page on AI features (last updated 10 December 2025) adds the retrieval side. AI Overviews and AI Mode may run several related searches for one question. The two features may use different models, so their answers and links vary.

The best public evidence on scale comes from SparkToro and Gumshoe's research, published 27 January 2026. Six hundred volunteers ran 12 prompts through ChatGPT, Claude and Google's AI, giving 2,961 runs. For ChatGPT and Google's AI, the chance of seeing the same list of brands twice was under 1 in 100. The same list in the same order came up about once in 1,000 runs.

The authors say the work was not peer reviewed and covers late 2025 only. Their conclusion still matters for buyers. They judged ranking position unreliable, and they judged the share of runs naming a brand a reasonable metric.

What is a visibility score, and how wide is the uncertainty?

A visibility score is a sample proportion: answers naming your brand divided by answers collected. Like any sample proportion, it comes with a margin that shrinks as the number of answers grows.

The table below shows that margin. It is our own calculation, run in Python on 11 November 2026, using the Wilson score interval at 95% confidence. The NIST statistics handbook describes the Wilson method as reliable across a wide range of sample sizes. It also never returns a lower limit below zero.

Answers collected (n)Observed 20%: plausible rangeObserved 50%: plausible range
105.7% to 51.0%23.7% to 76.3%
309.5% to 37.3%33.2% to 66.8%
10013.3% to 28.9%40.4% to 59.6%
30015.9% to 24.9%44.4% to 55.6%
1,00017.6% to 22.6%46.9% to 53.1%

Read the first row carefully. Two mentions in ten answers is shown as 20%, yet the true rate could be 6% or 51%. A dashboard that prints "20%" from ten answers is not wrong. It is just far less precise than it looks.

Two cautions apply. Precision improves slowly: you need roughly four times the answers to halve the range. And the interval assumes each answer is an independent draw. Ten answers to one prompt on one day are more alike than ten answers spread across prompts and days. Real uncertainty is therefore often wider than the table.

The interval also covers sampling error only. It says nothing about whether the right prompts were chosen.

Do API answers match what logged-in users see?

It is not publicly known how closely API answers match what a signed-in person sees. No AI vendor document we fetched compares the two. Treat any tracker's claim of an exact match as unproven unless it shows its evidence.

What can be sourced is limited. OpenAI's reproducibility guide describes its developer API, not the ChatGPT app. The SparkToro and Gumshoe study collected answers both by hand and through API calls. Its authors suspected API results might vary less. They reported that early data suggested this was not a major obstacle, and that they had not tested it thoroughly.

Personalisation and location add further gaps:

  • Location in ChatGPT. OpenAI's ChatGPT search help page says it collects general location from your IP address. It may share that with search providers to make results more accurate. The page does not say whether memory changes search results.
  • Personalisation in AI Mode. Google's AI Mode help page says past Search activity can tailor responses for signed-in adults with history switched on. Users can also connect Gmail, Calendar and Photos.

A tracker runs clean sessions with none of that history. Its score is best read as a baseline for an anonymous asker in the tracked location. How far a given customer's answer departs from that baseline is not published by any vendor. If you sell in the UK, check that the tool collects answers from a UK location. Our guide to showing up in Google AI Mode and Gemini covers Google's surfaces in more detail.

How do you tell a real change from noise?

A change is real when the gap between two periods is larger than chance alone would produce at those sample sizes. The standard check is a two-proportion comparison, which the NIST handbook sets out.

In plain English, the check works like this:

  1. Take the two rates, such as 20% last month and 30% this month.
  2. Pool both periods into one overall rate, as if nothing had changed.
  3. Work out how much two samples of that size would differ by luck alone.
  4. Divide the observed gap by that figure. A result above about 2 is unlikely to be chance.

Our own worked examples show why sample size decides the verdict. A rise from 20 of 100 answers to 30 of 100 scores 1.6. That is not enough to call. The same rise from 60 of 300 to 90 of 300 scores 2.8, which is a real change by the usual standard. A smaller rise, from 20% to 25% at 300 answers each, scores 1.5 and stays unproven.

A quick visual version: if the two 95% ranges barely overlap or do not overlap, the change is probably real. If one range sits mostly inside the other, wait for more data. For very small samples, NIST recommends an exact test instead.

What should you ask an AI visibility tracker vendor?

Ask how each number is produced, because the method decides how far you can trust it. A vendor that answers these seven questions plainly is easier to rely on than one with a prettier dashboard.

QuestionWhy it matters
How many answers sit behind each score?The table above shows the range at each sample size.
Do you show an interval or just a percentage?A bare percentage hides whether n was 10 or 1,000.
Do you query the API or the consumer app?The match between the two is not publicly established.
Which country and language are answers collected from?OpenAI says location can shape search results.
Are sessions signed in, with any history or memory?Personalised answers differ from clean ones.
Can I read every stored answer behind a score?You can audit a count. You cannot audit a summary.
How do you decide a change is real?Without a stated test, alerts may be reporting noise.

Be wary of "rank in AI" charts built from a handful of runs. The January 2026 SparkToro research found order was the least stable part of an answer. For prices and features across tools, see our comparison of AI visibility tools.

How to measure whether it worked

Measure with repeated runs on a fixed prompt set, compare periods with enough answers to separate change from noise, then check referral traffic as a second line of evidence. Neither number is enough alone.

  1. Fix the prompts and engines. Changing either mid-way gives you a different metric, not a trend.
  2. Collect a baseline of at least 100 answers per engine. At 20% visibility that gives a range of about 13% to 29%. Aim for 300 if you want to detect ten-point moves.
  3. Make your change, then collect the same number again. Spread runs across days, since same-day answers are more alike.
  4. Run the two-proportion check. Report both rates, both sample sizes and both ranges.
  5. Check traffic in GA4. Use the sources below.

For traffic, three documented facts help. OpenAI's publishers and developers FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs. That lets tools such as Google Analytics identify the visits. Google's GA4 traffic-source documentation says utm_source maps to the session source, so filter on "chatgpt.com".

Google's default channel group reference also lists an "AI Assistant" channel. It covers assistants such as ChatGPT, Gemini and Copilot. It excludes AI Overviews and AI Mode, which Google counts under Organic Search. Search Console likewise reports those two inside the "Web" search type, according to Search Central. We found no documented way to isolate AI Overview clicks in either tool.

Keep expectations modest. Referral data only counts people who clicked. A buyer who reads your name in an answer and searches for you later appears as something else. Visibility and traffic are separate measures, and they will not always move together.

CiteSieve, Braivex's tracker, takes the approach described here. It asks six AI engines a brand's customer questions once a day and stores every answer. Visibility is shown as the share of answers naming the brand, with the number of answers and a 95% interval. For manual methods, see how to track brand mentions in ChatGPT.

Questions and answers

Is a daily AI visibility score reliable?

A single day's score is usually built from few answers, so its range is wide. By our calculation, ten answers showing 20% could reflect a true rate anywhere from about 6% to 51%. Daily figures are useful as raw data. Judge trends on weekly or monthly totals, where the answer count is large enough to narrow the range.

Can a tracker make AI answers consistent by setting temperature to zero?

Not reliably. Anthropic's documentation says outputs are not fully deterministic even at temperature 0. OpenAI describes seeded outputs as only mostly deterministic. Google says its newest Gemini models ignore custom temperature values. Fixed settings would also stop the tracker resembling normal use, which is the thing you are trying to measure.

Should I track my ranking position in AI answers?

Treat position with caution. SparkToro and Gumshoe's research, published 27 January 2026, found the same list in the same order appeared about once in 1,000 runs. One brand in that study appeared in 69 of 71 ChatGPT answers but came first in only 25. How often you appear is the steadier measure.

Does traffic from Google AI Overviews show up separately in GA4?

No. Google's default channel group reference places AI Overviews and AI Mode traffic under Organic Search, and its AI Assistant channel excludes them. Search Console also reports them within the Web search type. We found no Google documentation describing a way to separate AI Overview clicks from other organic clicks in either tool.

Sources

  1. OpenAI API docs: Advanced usage (reproducible outputs)
  2. Anthropic Claude docs: Glossary (temperature)
  3. Google AI for Developers: Gemini API prompting strategies
  4. Google Search Central: AI features and your website
  5. OpenAI Help Center: Publishers and developers FAQ
  6. OpenAI Help Center: ChatGPT search
  7. Google Search Help: AI Mode
  8. Google Analytics Help: Default channel group
  9. Google Analytics Help: Traffic-source dimensions and campaign parameters
  10. SparkToro and Gumshoe: AIs are highly inconsistent when recommending brands or products (27 January 2026)
  11. NIST/SEMATECH e-Handbook: Confidence intervals for a proportion (Wilson method)
  12. NIST/SEMATECH e-Handbook: Do two processes have the same proportion of defectives?

Sources were read on 11 November 2026. Engines and their documentation change; check the linked page for the current wording.

More guides

Find out what AI says about your brand.

A site audit on the page in seconds, then live answers by email. No account, no card.