SEO for Large Websites: A Step-by-Step Guide to Index Tiers, Crawl Waste, and Template QA
A practical field guide for sites with tens of thousands of URLs: how to segment your estate, stop crawl budget leaks, and ship template fixes that stick.
SEO for Large Websites Is a Segmentation Problem Before It Is a Content Problem
Picture an in-house head of search at a classifieds or property portal with roughly 250,000 URLs. Every two-week release ships new filter combinations and a redirect that points to another redirect, and the search team only finds out three weeks later when Search Console flags a crawl anomaly. No amount of new content fixes that. The site has never been segmented: Googlebot crawls archived listings and parameter permutations as eagerly as the pages that sell.
This guide treats SEO for large websites as a site-level engineering discipline: deciding which URLs deserve attention, stopping waste at the template layer, and closing the loop with crawl data so fixes stick.
Crawl budget is not a universal concern. Google states it only meaningfully constrains very large sites (roughly a million-plus pages) or sites whose URL sets grow or change rapidly. A 250,000-URL portal with faceted filters sits in the second category; a 3,000-page brochure site does not.
What Has to Be in Place Before You Touch a Large Site
- A full URL inventory pulled from sitemaps, server logs, and a crawl - reconciled into a single list.
- Search Console access for every property and subdomain, plus log-file access from your infrastructure team.
- A staging environment with a crawler you can run against template changes before release.
An indicative large-site technical audit from an agency typically runs R40,000-R120,000 in South Africa depending on URL volume and log access. Treat that as a scoping estimate, not a quote.
Step 1: Cut the URL Estate Into Index Tiers (and Write the Rules Down)
Assign every URL class - not every single URL - to one of three tiers, then write the rule that governs it.
| Tier | What lives here | Rule |
|---|---|---|
| Tier 1 | Money pages: category and location hubs, key listings | Index, self-canonical, in sitemap, linked from nav |
| Tier 2 | Long-tail support, entity and use-case pages | Index where unique value exists, else canonical |
| Tier 3 | Facets, internal search, archive, empty categories | Noindex or robots-block, excluded from sitemaps |
Where Crawl Budget Actually Bites (and Where It Doesn't)
Google describes crawl budget as crawl capacity (how much Googlebot can fetch) combined with host load limits (how much your server tolerates). It becomes a real problem when your URL set changes faster than Google can re-crawl it - every day on a portal where listings expire, rarely on a stable marketing site.
Step 2: Get Faceted Navigation and Internal Search Under Control
Facets are where large sites leak most. Colour × size × price × sort creates combinatorial loops Googlebot will follow until it times out. Index only facet combinations with demonstrable query demand - evidence from Search Console queries, not instinct - and manage the rest with canonical, noindex, or robots rules. Block internal search results pages outright; they add no crawl value and often duplicate your tier 1 pages.
Index, Canonical, or Block? A Decision Table for Facet and Parameter URLs
| URL type | Action | Why |
|---|---|---|
| Facet with query demand | Index, self-canonical | Earns impressions on its own |
| Multi-facet permutation | Canonical to parent | Preserves link equity, avoids loops |
| Sort, session, tracking params | Robots-block or noindex | Zero search value |
| Internal search results | Noindex + robots-block | Crawl drain, no unique content |
