LLM Visibility: The Repeatable Routine for Measuring Brand Mentions in ChatGPT, Claude and Gemini
A practical how-to for building a locked prompt library, scoring mention and citation rates, and avoiding the sampling errors that quietly corrupt AI visibility data.
LLM visibility is not something you check once. It is a rate you measure across a fixed set of prompts, at a recorded model snapshot, over time. This guide shows you the routine.
Ask ChatGPT whether it recommends your business and you will get an answer. Ask again tomorrow and the answer may differ. That variability is not a glitch - it is the core measurement problem of LLM visibility, and almost every team gets it wrong on their first attempt by treating a single response as evidence.
LLM Visibility Is a Sampling Problem Before It Is a Marketing Problem
Large language models generate text probabilistically. OpenAI's own documentation describes sampling parameters such as temperature that directly control how much randomness enters each response. Even at low temperature, model providers ship updates, routing changes between model variants, and personalisation layers. The practical consequence: the same prompt on the same day can return different brand mentions across two sessions.
This means visibility must be treated like a survey, not a lookup. You define a sample (a fixed prompt library), hold the conditions as constant as you can (model, version, settings, date), and compute a rate. "We appeared in 9 of 40 prompts this fortnight" is a measurement. "I asked and it didn't mention us" is an anecdote.
prompts is a workable starter library - large enough for a stable rate, small enough to run fortnightly by hand
Why "I Asked ChatGPT and It Didn't Mention Us" Is Not a Data Point
Consider a property agency in Pretoria testing whether ChatGPT names it for "bond originator in Pretoria". One run returns three national banks and no agency. That single answer proves nothing about the agency's actual position. It may have been a low-temperature run where the model favoured large entities; the next session may surface local agencies. Without repeat runs across a prompt set, there is no way to distinguish a real coverage gap from sampling noise.
The same applies to ecommerce. A retailer testing "best load-shedding inverter under R5 000" against a single response cannot tell whether the absence of its brand reflects weak content, a model that defaults to marketplace listings, or simple variance. Only a rate across many prompts, recorded at a fixed snapshot, answers the question.
A visibility score with no model version and date attached is not comparable to anything. File it as an anecdote, not a baseline.
Prerequisites: Four Things to Lock Before You Run a Single Prompt
Before any measurement run, lock these four variables. Skipping any one of them invalidates every comparison you make afterwards.
| Prerequisite | What to lock | Why it matters |
|---|---|---|
| Model snapshot | Exact model name and version, plus any account-level settings | Providers update models; scores drift without a documented snapshot |
| Prompt library | 30-60 real customer questions, clustered by intent | A stable sample is the only basis for a comparable rate |
| Scoring rubric | Definitions for mention, citation, sentiment and accuracy | Two reviewers must score the same answer identically |
| Cadence | Fortnightly or monthly, with a named owner | Trends need repeated samples; one-off audits hide movement |
Step 1: Build the Prompt Library from Real Buyer Questions
Do not invent prompts from keyword tools alone. Pull them from sales call notes, live chat transcripts, support tickets and site search logs. Group them into intent clusters - for example: discovery ("who does X in [city]"), comparison ("X vs Y"), price-led ("X under R5 000"), and problem-led ("how do I fix X"). Aim for balanced coverage across clusters so one noisy cluster cannot dominate your aggregate rate.
Write prompts the way customers speak, not the way marketers write. A clinic checking Perplexity for its service query should use the phrasing patients use on the phone, not the phrasing on the service page. Keep the library versioned in a spreadsheet or document, with a change log. When you add or retire prompts, note the date - it changes your denominator.
Step 2: Capture a Baseline Snapshot with Model, Version and Date Recorded
Run the full library once, in a single session, and save the raw output. Record the engine, the model version, the date and time, and any settings that could affect sampling. For engines with live-search behaviour, note whether citation sources appeared in the answer.
run_id: 2024-11-04_a
engine: chatgpt
model: [record exact version string shown in product]
settings: [temperature / account defaults if visible]
library_version: 1.0 (42 prompts)
clusters: discovery 12 | comparison 10 | price-led 10 | problem-led 10
scoring: mention / citation / sentiment / accuracy
run_by: [name]
prompt_07 | cluster: price-led
text: "best load-shedding inverter under R5 000"
mentioned: no | cited_url: none | sentiment: n/a | accuracy: n/aThis snapshot is now your reference point. Everything you measure later is compared against it - which is precisely why the model version and date must be in the record. If the provider ships a new model between runs, you are comparing across a changed system, and your report should say so.
Step 3: Score Every Answer on Mention, Citation, Sentiment and Accuracy
Score each response against your rubric. Four fields, each with a narrow definition, so two people scoring the same answer reach the same result:
- Mention - did the answer name your brand at all? Binary, yes or no. Aggregate into a mention rate.
- Citation - did the answer link to or reference a URL you own? For live-search engines, this is often more actionable than a mention, because it shows the model actually retrieved your content.
- Sentiment - was the brand framed positively, neutrally or negatively? Keep this to a simple three-point scale to stay consistent.
- Accuracy - were factual claims about your business correct? A mention with a wrong price or service list is worse than no mention.
Report the results as rates, not counts: mention rate = mentions ÷ prompts run; citation rate = cited answers ÷ prompts run. Rates survive changes to library size, counts do not. Keep raw outputs alongside scores so you can audit a disputed result later - and so a reviewer can check whether a "no" was a genuine absence or a run where the model simply paraphrased three competitors.
If you automate collection, save the full response text, the timestamp and the engine name in a structured store. A scored spreadsheet without the raw answer cannot be re-checked when the model changes.
