SERP Data: How to Capture, Store and Validate Search Results You Can Defend
A practical how-to for SEO engineers and analysts who need a trustworthy SERP dataset - capture conditions, schema, reconciliation and the mistakes that quietly corrupt a benchmark.

You have inherited twelve months of rank-tracking exports from someone who has left the company. The marketing director wants one number: did the campaign work? Your export says positions improved by 4.2 places. Search Console says average position got slightly worse. Both cannot be true, and right now you cannot explain why either is.
That gap is what separates a SERP dataset from a folder of CSVs. This guide treats search results as a measurement instrument: how to define it, capture it under controlled conditions, store it so six months from now you still know what it means, reconcile it against data you already trust, and avoid the errors that quietly poison a benchmark.
SERP Data Is an Asset You Build, Not a Screenshot You Take
A defensible dataset has three properties: every row carries the conditions it was captured under, every query has a fixed identity over time, and every derived metric traces back to specific capture dates. If you cannot answer "on what device, in what location, on what date, with what personalisation context" for a number, that number is an anecdote.
Google states plainly that results are personalised by factors including location and search history. Any benchmark that does not record those conditions is measuring a different SERP for every analyst who opens it.
SERP API, Rank Tracking, Scraping, Visibility Data: Naming the Thing You Actually Need
| Term | What it actually returns | Defensible for |
|---|---|---|
| SERP API | Raw result stack per query, including feature blocks and source URLs | Feature ownership, AI citation audits |
| Rank tracker | Position per keyword, usually one device and one locale | Trend monitoring, not feature analysis |
| SERP scraping | Parsed HTML from search pages at your own cadence | Volume capture, subject to fragility and drift |
| Search visibility data | Aggregated share-of-voice metrics across a keyword set | Executive reporting, not diagnosis |
Most disputes arise because someone compares a rank-tracker position with a visibility index and assumes they measure the same thing.
Prerequisites: Five Decisions to Lock Before You Capture a Single Result
- Query set and version number
- Locale and location granularity (country vs city - for South African property clients, city-level matters)
- Device matrix: mobile only, or mobile plus desktop
- Capture cadence: weekly, fortnightly, monthly
- Owner of the data contract, so schema changes go through one person
Step 1 - Define and Freeze the Query Set
Version the list. Adding keywords mid-year invalidates any comparison to prior months unless you segment by version. Three examples of queries that belong in a defensible set:
- An ecommerce category term with a qualifier: waterproof hiking boots wide fit
- A Cape Town property portal term: two bedroom apartments for sale sea point
- A local service term: emergency geyser repair johannesburg
Each is specific, has a plausible buyer behind it, and its conditions will differ materially by location.
Step 2 - Choose Capture Conditions and Store Them With Every Row
A schema keyed on the query alone is not enough. Minimum viable row structure:
CREATE TABLE serp_capture (
query_id TEXT, -- stable hash of the query string
query_version SMALLINT,
locale TEXT, -- e.g. en-ZA
location TEXT, -- e.g. Cape Town, ZA
device TEXT, -- mobile | desktop
captured_at TIMESTAMPTZ,
position SMALLINT,
feature_type TEXT, -- organic | ai_overview | local_pack
source_url TEXT,
cited_in_ai_answer BOOLEAN
);
Treat a URL cited inside an AI Overview or AI Mode answer as a citation, never as a ranking. Storing a citation in the position column is the single most common corruption we see.
Step 3 - Choose a Data Source and Size the Request Budget
Work the arithmetic before negotiating with anyone:
5,000 keywords x 4 captures/month x 3 devices
= 60,000 requests/month
At an estimated USD 3-15 per 1,000 requests:
= USD 180-900/month (~ZAR 3,300-16,500 at R18.40/USD, estimate)
South African teams generally bill in USD, exposing the budget to FX swings - model a 10% buffer. If that figure is uncomfortable, cut cadence to fortnightly before you cut the device matrix; a consistent monthly mobile baseline beats an erratic three-device one.

