You published a page, but why can't you find it on Google? Sometimes a page is indexed and still gets no search traffic. To understand why, separate three jobs: discovering and crawling pages, indexing them, and choosing results when someone searches.
This guide uses Google as the example. It explains what happens at each stage and how to check your own page.
How Search Engines Work in Three Stages
A search engine does not browse the entire web from scratch each time you type a question. It gathers and processes pages beforehand, then searches its index for suitable results.
| Stage | What happens | What to ask |
|---|---|---|
| Crawling | The engine discovers a URL and attempts to retrieve its content. | Does Google know this URL, and can it access it? |
| Indexing | It analyzes the page and decides whether to store information about it. | Has Google understood and indexed this page? |
| Serving results | For a particular search, it selects, orders, and displays results. | Does this page meet the searcher's current need? |
Discovery does not guarantee indexing. Indexing does not guarantee a prominent position for every query. Diagnose the stages separately.
How Does Google Discover Pages?
Googlebot is Google's crawler. It can discover a new URL by following a link on a page it already knows. For example, if an auto parts retailer adds a brake-pad product page and links to it from a category page, the crawler has a path to the new page. This is one reason SEO includes making important pages accessible.
A sitemap can list important URLs and help Google discover them. But listing a URL does not guarantee that Google will crawl or index it. Clear internal links still help people and crawlers reach important pages.
After finding a URL, Google must retrieve the content. Login restrictions, server errors, or crawl-blocking settings can prevent access. If important content loads through JavaScript, Google may also need to render the page. Opening it successfully in your browser does not prove that a crawler sees the same content.
What Is Crawl Budget?
Crawl budget describes the set of URLs Google can and wants to crawl on a site within a period of time. Server capacity and Google's demand to revisit the pages both matter.
On a large site, many duplicate or unimportant URLs can consume crawling resources before important pages are found. A smaller site with normally updated pages usually does not need a special crawl-budget project. Start with reachable pages, a current sitemap, and a reliable server.
Does Crawling Guarantee Indexing?
No. Google analyzes the page's text, title, images, and other information to understand its subject and decide whether to store it in the index. A successful crawl is a prerequisite, not an indexing promise.
The brake-pad product page from our example still needs an indexing decision after it has been crawled. If vehicle filters create several near-identical URLs for that product, Google may choose one as the main version, known as the canonical URL. The alternate URLs not being indexed separately does not necessarily mean something is broken.
A site owner can also prevent indexing. A noindex directive tells Google not to add a page to its index; robots.txt primarily controls whether crawlers may access a URL. They are not interchangeable: if robots.txt blocks the page, Google may never see a noindex directive on it.
When a page is reported as not indexed, first identify whether access failed, Google selected another version, indexing was deliberately blocked, or Google processed the page without choosing to index it. Repeatedly submitting the same URL does not diagnose the problem.
Why Does the Same Page Perform Differently Across Searches?
When someone searches, Google looks in its index for results that might answer that query and orders them for that search. It considers the page as well as context such as the query, location, language, and device. The complete ranking calculation is not public.
The same brake-pad product page might suit a search for “brake pad price for my car model.” But “why are my brakes squeaking” calls for an explanation and troubleshooting steps. Both searches concern brakes, but they need different kinds of pages. That difference is search intent.
That is why a page has no permanently fixed ranking. Freshness may matter more for prices or news than for a stable definition. Location, competing pages, and search-system changes can also affect results. Improving visibility begins with answering the intended search, not repeating its keyword more often.
Why Are Search Results More Than Webpage Links?
Google also changes how it presents results to fit the question. A product search may show shopping information; a search for a nearby auto repair shop may show a map and local businesses; an explanatory query may show a snippet, images, or an AI-generated answer. Ads can appear too, but an ad position is separate from organic ranking.
Before judging a page's performance, read the actual results page for its target query. If those results mainly show local businesses, a broad blog post may not satisfy someone ready to book a repair. That is a page-type problem, not something solved by adding a few more keywords.
How to Check Where Your Page Is Stuck
Choose one important page and enter its full URL in Google Search Console's URL Inspection tool. Then check in order:
- Discovery: Does Google know the URL? If not, check whether a relevant category page or article links to it and whether the sitemap lists the correct address.
- Crawling: Can Google access it? Check crawl permissions, the page-fetch result, and whether important content can be read.
- Indexing: Is it indexed? If not, read the specific reason. If the report shows Google's selected canonical URL, check that too.
- Search performance: Once indexed, use the Performance report to see which queries generate impressions and clicks. If traffic is absent, compare the page with the actual search results and the user's intent.
URL Inspection initially shows the information Google has stored about the page. After a change, “Test live URL” checks the current version. Passing that test means the tool can access the page; it does not guarantee indexing or ranking. You can request indexing after fixing a problem, but repeating the request will not make the same URL crawl faster.
Frequently Asked Questions
How Do Search Engines Work Step by Step?
They discover and crawl a page, analyze it, and decide whether to index it. When someone searches, their systems select and display relevant results from the index. Passing one stage does not guarantee passing the next.
Do Search Engines Use AI?
Yes. Some Google systems use AI to help understand queries and page content. An AI-generated summary is one visible application, but one AI model does not single-handedly decide every search ranking.
Why Was My Page Crawled but Not Indexed?
Crawling means Google accessed the page; it does not guarantee indexing. “Crawled – currently not indexed” means Google has not added it to the index yet, although it may do so later. Inspect the URL and check for near-duplicate versions; repeatedly requesting a crawl will not make it happen faster.