Crawling is a search engine sending a program to visit your page and read what’s on it. Indexing is what happens next: the engine works out what the page is about and decides whether to store it in the database it searches when someone types a query. These are two separate stages with two separate ways to fail, and both must succeed before your page can compete for anything. Google states plainly that indexing isn’t guaranteed.
In plain English
Think of postal collection and a sorting office.
A driver runs a fixed round each day, emptying the boxes on their route. That’s crawling. What comes back goes to a sorting office, where someone decides what each item is, whether it duplicates something already handled, and whether it gets processed at all. That’s indexing.
Two failures are possible and they look identical from your side. The driver never reached your street, or the driver collected your letter and the sorting office set it aside. In both cases nothing arrives, and the fix for one is useless for the other.
Why this matters for your site
Almost every serious search problem in technical SEO, the work of making a site reachable and readable by search engines, lives at one of these two stages. Yet it usually gets reported as a ranking problem, meaning a problem with the order results appear in.
The distinction matters because the responses are opposite. A page that isn’t being crawled needs an access or discovery fix. A page that’s crawled and not indexed needs a quality or configuration fix. Buying content for the first one, or opening up crawling for the second, wastes months.
How it actually works
Discovery. There’s no central register of web pages, so Google keeps looking. Most pages are found through links from pages it already knows. You can also submit a sitemap, a file listing the addresses on your site you care about.
Crawling. Google’s crawler is called Googlebot. It decides algorithmically what to visit and how often, and deliberately slows down to avoid overloading a server. It won’t crawl pages that require a login, and it respects rules that block its access.
Crawl budget. Google defines this as the set of URLs it can and wants to crawl. Two halves. Capacity is what your server can take: Google says every site starts on the same default, conservative limit, which rises over time if there’s demand and the site stays healthy, and falls when responses slow down or the server returns errors. Demand is how much Google wants your pages, which for Search it weighs using popularity, overall user value, content uniqueness, and serving capacity. Google added a detail in July 2026 that matters for large sites: capacity is shared across all of its crawlers, so heavy demand from one reduces what’s left for the others.
Indexing. Google analyzes the text, the title, image descriptions, and more. It groups near-identical pages and picks one to represent the group. Then it decides whether to store the result. Google names three common reasons this fails: the content quality is low, robots rules in the page code disallow indexing, or the site’s design makes indexing difficult.
Mobile-first indexing. Google evaluates and indexes the version of your page it sees on a phone. If content exists on your desktop layout but not the mobile one, treat it as content Google doesn’t have.
What this looks like in practice
An online retailer adds filters for size, color, and price. Each combination produces its own web address, and the site now generates several hundred thousand of them.
Googlebot spends its round on filter combinations. New product pages take six weeks to appear instead of two days, and some never do. Nothing is broken, no error appears anywhere, and the site is faster than it was last year.
This is the crawl budget case in its usual form. The problem isn’t that Google is unwilling; it’s that the round is being spent on addresses nobody searches for.
What people usually get wrong
“Crawled means indexed.” They’re separate stages, and Search Console reports them separately for exactly that reason.
“We need to fix our crawl budget.” For most sites there’s nothing to fix. Google frames it as a concern for large sites and sites that generate many URLs automatically.
“We blocked it in robots.txt, so it’s out of Google.” Blocking crawling and blocking indexing are different controls. Google says a blocked address can still appear in results. To keep a page out, allow crawling and use a noindex rule in the page code.
“Speed up the server and Google will crawl more.” Only up to a point. Google says that if demand is low, it crawls less regardless of available capacity.
What you can do about it
Open the page indexing report in Google Search Console and read the exclusion reasons. This is the fastest honest answer to what Google holds and what it doesn’t.
Run URL Inspection on an important page and look at the HTML Google received. If your main content is missing from it, the crawl succeeded and the reading didn’t.
Check the crawl stats report for response times and error codes. A rise in server errors here explains a lot of otherwise mysterious declines.
Don’t block the CSS and JavaScript files your pages need to display. Google needs the same resources a browser needs.
On a large site, the question of which of your pages Google actually holds is rarely answerable from a dashboard alone, and it’s one of the things an SEO audit is built to establish.
Words you will hear
Crawling. A search engine visiting a page and reading what is there.
Googlebot. The program Google uses to do it.
Indexing. Analyzing a page and storing it in the database results are drawn from.
Crawl budget. The set of URLs Google can and wants to crawl on your site.
Mobile-first indexing. Google indexing the version of a page it sees on a phone.
For the index, rendering, robots.txt, sitemap and canonical, see How search works. For SERP, query and search console, see What is SEO?
Technical SEO is the work of making sure a search engine can reach your pages, read them, and store them. Failing at it guarantees invisibility; clearing it guarantees nothing. It sits underneath everything else in SEO, the wider practice of making a site findable in search, and Google publishes a minimum bar for it: Googlebot isn’t blocked, the page works, and the page has content Google can index.
In plain English
Technical SEO is the wiring and plumbing of a building.
Nobody visits an office because the electrical work is good. They come for what happens inside. But a fault in the wiring is invisible from the street and total in effect: when the water goes off, the building empties, and no amount of redecorating brings anyone back.
That’s the shape of every technical problem on a website. While one lasts, nothing else you do matters, and it’s usually silent, usually caused by a change somebody made for an entirely unrelated reason.
Why this matters for your site
Content problems cost you some traffic. Technical problems cost you all of it.
They’re also the hardest to notice from the inside, because your site works perfectly for you. You’re logged in, your browser has cached half the site, and you aren’t a crawler. A search engine arrives from a different network, without a session, running a program rather than a browser, and it can be handed something completely different from what you see.
That’s why traffic sometimes falls off a cliff a few weeks after a redesign, a hosting move, or a security update, and why the cause is rarely where people look first.
How it actually works
Google publishes exactly three technical requirements for a page to be eligible for indexing, meaning stored in the database Google searches when someone types a query.
- Googlebot isn’t blocked. Googlebot is Google’s crawler, the program that visits and copies pages. Google only indexes pages that are publicly accessible, so anything behind a login is out, and so is anything blocked in robots.txt, a file at the root of a site that tells crawlers where they may go. One useful detail: blocking a page in robots.txt stops crawling, but the address may still appear in results. To keep a page out of the index properly, allow crawling and use a noindex rule in the page’s code.
- The page works. Google only indexes pages served with an HTTP 200 status code, which is the server’s way of saying the request succeeded. Error responses aren’t indexed, whether the visitor’s request caused them or the server did. You can check the code for any page with the URL Inspection tool in Google Search Console.
- The page has indexable content. The text has to be in a file type Google supports, and the content mustn’t violate Google’s spam policies.
Then the part everyone skips. The three requirements are a floor, not a promise: Google says plainly that meeting them doesn’t mean a page will be indexed. Indexing isn’t guaranteed.
What technical SEO covers beyond the floor
Most technical work is about the second-order versions of those three questions. In rough order of how often they cause real damage:
- Crawling and indexing management. Which of your pages get visited, how often, and which ones end up stored.
- Robots.txt and sitemaps. The file that controls crawler access, and the file that lists the addresses you want found.
- Redirects. Instructions that send one web address to another, and the single most common cause of traffic loss during a site move.
- Duplicate content and canonical tags. When the same content sits at several addresses, a canonical tag tells search engines which one to treat as the real one.
- Error pages. The difference between a page that’s gone permanently, one that’s missing temporarily, and one that claims to work while showing nothing.
- JavaScript rendering. Google renders pages and runs their JavaScript using a recent version of Chrome. Sites that build their content this way can still go wrong in ways that plain HTML sites can’t.
- Structured data. Extra information in a page’s code describing what the page contains, which can make richer results possible.
- Core Web Vitals. Google’s measurements of loading, responsiveness, and visual stability.
- Multilingual and multi-country setup. Telling search engines which version of a page belongs to which language and market.
What this looks like in practice
A retailer adds bot protection after a spike in scraping. The setting is sensible and the site gets faster for real visitors.
Googlebot now receives a verification page instead of the product page. That verification page returns an HTTP 200 success code, because from the server’s point of view nothing failed. So the pages stay in the index. They just contain nothing.
This is the standard shape of a technical problem, correct-looking on every dashboard and fatal underneath. Rankings decay over six weeks with no error in any report and no obvious cause, and nobody connects the security setting to the traffic loss, because a firewall isn’t an SEO decision.
What people usually get wrong
“We did technical SEO last year.” Every deployment is a chance to break it. Technical health is a monitoring problem, not a project.
“The site loads fine for me.” You aren’t the crawler. Check what Googlebot receives, using the URL Inspection tool, rather than what your browser shows.
“Technical SEO means site speed.” Speed is one part of it, and rarely the part that’s actually broken. Reachable and readable come first.
“We blocked it in robots.txt, so it’s out of Google.” Google says blocked addresses can still appear in results. Blocking crawling and blocking indexing are two different controls.
What you can do about it
Open the page indexing report in Google Search Console and read the reasons pages are excluded. It’s free, needs no technical skill, and answers most questions before anyone guesses.
Run URL Inspection on one important page and look at the HTML that Googlebot actually received. If your product description is missing from it, you’ve found something.
Open yourdomain.com/robots.txt in a browser. It takes ten seconds, and occasionally it explains everything.
The gap between cause and visible effect is usually weeks, which is exactly why the two never get connected. So treat every launch, migration, hosting change and security change as a reason to check indexing a week later.
Know where your line is. A wrong rule at this level doesn’t degrade performance, it removes a site, and undoing that takes longer than causing it. Server responses, redirect rules and indexing directives are developer work.
Words you will hear
HTTP status code. The server’s short answer about what happened with a request. 200 means success.
Googlebot. The program Google uses to visit and copy pages.
Staging site. A private copy of a website used for building and testing before changes go live.
URL Inspection tool. The part of Search Console that shows what Google received and stored for a single page.
CDN. A content delivery network, a service that serves your site from servers closer to the visitor. It sits between your site and every crawler.
For search console, query and SERP, see What is SEO? For crawler, index, rendering, robots.txt, sitemap and canonical, see How search works.
