Search engines crawl webpages, store information about them in an index, and choose relevant results when someone searches.
When a published page is missing from Google, the cause could be discovery, access, indexing or its fit for that search. We will follow one brake-pad product page through these stages, then use Search Console to check it.
How Search Engines Work in Three Stages
A search engine does not browse the entire web from scratch each time you type a question. It collects and processes pages beforehand. Its index is a searchable database of information about those pages.
| Stage | What happens | What to ask |
|---|---|---|
| Crawling | The engine discovers URLs and retrieves page content. | Can Google find and access this page? |
| Indexing | It analyzes the content and decides what to store in its index. | Has this page been included, or has another version been chosen? |
| Serving results | It selects, ranks and displays results for a particular search. | Does this page answer what the searcher needs? |
Being discovered does not guarantee indexing, and being indexed does not guarantee a high ranking. These are separate stages, so a missing page and a low-ranking page need different checks.
How Does Google Discover Pages?
Googlebot is Google's web crawler: software that visits pages and follows links. If an auto parts retailer links to a new brake-pad product from its brake-parts category, Google has a route to discover that product page. Clear internal links help customers reach it too.
A sitemap lists important URLs for Google to discover. Keep it updated as products are added or removed; submitting it does not guarantee a crawl or indexing.
To read the product details, Google needs access to the page. A login requirement, server error or crawl-blocking rule can prevent that. Google also renders pages—running their code to display the content—because some sites load product details through JavaScript. A page opening in your own browser does not prove that a crawler can read the same information.
What Is Crawl Budget?
Crawl budget describes how much Google can and wants to crawl on a site. It depends on crawl capacity, including how much the server can handle, and crawl demand: how useful it is for Google to fetch or revisit those URLs.
An auto parts catalogue can generate many similar URLs through vehicle, brand and sort filters. At scale, crawling these variants can delay visits to more useful pages. A small site usually needs clear links, an up-to-date sitemap and a reliable server before it needs a separate crawl-budget project.
Does Crawling Guarantee Indexing?
No. During indexing, Google analyzes a page's text, title, images and other information to understand it and decide what belongs in the index.
For the brake-pad page, that includes deciding whether it is a distinct product page or another version of content already found. If filters create near-identical URLs for the same product, Google may choose one representative version: the canonical URL. The other versions not being indexed separately can be normal.
Site settings matter too. A noindex directive tells Google to exclude a page from search; robots.txt controls crawling. Blocking crawling does not reliably remove a URL from search, and it can stop Google reading a noindex directive on the page.
Why Does the Same Page Perform Differently Across Searches?
When someone searches, Google's ranking systems compare results from its index. They assess relevance and usefulness using many signals, including the content and links, alongside context such as the searcher's location, language and device. There is no publicly available formula that lets you calculate a page's final position.
Our brake-pad product page could answer “brake pad price for my car model” if it provides compatibility, price and availability. Someone searching “why are my brakes squeaking” needs an explanation of possible causes and what to check. The topic overlaps, but the task and suitable page type differ. That is search intent.
A page has no permanently fixed ranking. The query, competing pages and search context can change the results. Freshness matters more for searches needing current information, such as breaking news, than for a stable definition.
Why Are Search Results More Than Webpage Links?
Google's results can include product listings, maps, images, videos and AI-generated answers, depending on the search. Ads may appear alongside these results, but buying an ad does not improve a page's organic ranking.
Before deciding what to improve, look at the results page for the search you want to answer. A query about buying brake pads may lead to product pages; one about nearby brake repair may favor local businesses.
How to Check Where Your Page Is Stuck
Use Google Search Console, Google's free tool for checking how your site appears in search. First add and verify your website, or ask its owner for access. Select its property—the website entry you manage in the tool.
Paste the full product-page URL into the inspection bar at the top, and open Page indexing. The initial result describes Google's saved information, not a new check of the live page.
- Check discovery and crawling. If the URL is unknown to Google, confirm that the product category links to it and the sitemap lists it. “Discovered – currently not indexed” means Google knows the URL but has not crawled it yet. If fetching failed, address the reported access or server error. Check the last crawl date before assuming an old error still exists.
- Read the indexing reason. An accidental noindex on a public product page needs removing. Check Google-selected canonical where available. If it points to another URL, inspect that page: a near-identical filter variant may not need separate indexing.
- Test the current page after a fix. Choose Test live URL, then inspect the returned HTML under View tested page for the product name and compatibility details. A successful test is not confirmation that the page has been indexed. After a meaningful fix, you can request indexing; repeatedly submitting the same URL will not speed up crawling.
- Check search performance once indexed. Open Performance → Search results, choose Web search and the last three months. If Web is split into text-based and multimodal views, choose text-based. In Pages, select the product URL to filter the report, then open Queries. Select the Google-selected canonical URL, which receives most performance data for the page. Impressions record appearances in results; clicks record clicks on those results, not confirmed page loads. Our Search Console guide explains how to read the report.
For example, an indexed product page with impressions but no clicks needs a different investigation from a page Google cannot fetch. Compare its queries and search results with the product information it offers before rewriting it. An empty performance report alone does not prove a page is unindexed; use URL Inspection for that check.
Frequently Asked Questions
How Do Search Engines Work Step by Step?
Crawlers discover and fetch pages. The engine processes their content for its index. When someone searches, its systems choose and order relevant results. Google also revisits pages to find changes, so this is an ongoing process.
Do Search Engines Use AI?
Yes. Google uses AI in systems that help interpret queries and page content. AI-generated answers are one visible application; ordinary search rankings also involve multiple systems, rather than one AI model deciding everything.
Why Was My Page Crawled but Not Indexed?
Crawled – currently not indexed means Google fetched the page but has not added it to the index. The status alone does not identify the cause. Check whether the page adds useful, distinct information and whether another version covers the same need. It may be indexed later, but resubmitting an unchanged page is not a fix.

