A phone-repair website has public service pages and a set of internal search-result URLs. The owner wants search engines to spend less time requesting the search results without hiding the actual repair services. Is robots.txt the right tool?
Robots.txt is a crawl-access file. It tells compliant crawlers which URL paths they may request on a particular site. It does not delete pages, keep private information safe, or guarantee that a blocked URL will vanish from search results. Choose the file for crawl access, not for secrecy or index removal.
What Does robots.txt Actually Control?
When a crawler arrives, it looks for a file at the root of the host it is visiting, such as https://example.com/robots.txt. The file can name a crawler with User-agent and set Disallow or Allow paths. Allow can make a narrower exception to a restriction; neither directive decides whether a page belongs in the index. A matching Disallow asks that crawler not to fetch those URLs. It does not stop people opening them in a browser.
Rules apply to the exact protocol, host, and port serving that file. A file on www.example.com does not automatically control shop.example.com. This is one reason to test the real URL, not just read a rule that looks similar.
How Do You Read a Small robots.txt File?
User-agent: *
Disallow: /internal-search/
Sitemap: https://example.com/sitemap.xmlHere, * addresses crawlers generally. The Disallow line asks them not to fetch URLs under /internal-search/. The Sitemap line points to a preferred-URL list; it is not an Allow rule. For what belongs in that list, see the XML sitemap guide. This is an illustrative file, not a copy-and-paste recommendation for every site.
Be exact about paths. Disallow: / blocks crawling across that host for the matching crawler. Disallow: /internal-search/ is narrower than Disallow: /internal-search, which can also match other paths beginning with those characters. A small typing change can block more than intended. If an Allow and Disallow both match, Google uses the more specific path rule; do not assume the last line wins.
What Can robots.txt Not Do?
- It cannot reliably remove a webpage from Google Search. Google may know a blocked URL from links and show the address without having fetched its contents. If you need a public page excluded from Google's index, use
noindexwhile allowing Google to crawl and read it. - It cannot protect confidential data. The file is public, and crawlers need not obey it. Use authentication or server-side authorization for private information.
Do not block a URL in robots.txt and then expect Google to read a noindex tag on that same page. The block can prevent the crawl needed to see the tag. For the decision about which pages belong in search, read indexing and noindex; for the wider process, see technical SEO basics.
When Might a Rule Be Useful?
Use a rule when you have a specific crawl-access problem: for example, a measured crawl problem involving many low-value internal search URLs. Before blocking a folder, list the URLs it contains. Keep important service and product pages—and files needed to render their main content—crawlable. Many small websites do not need special rules at all.
Keep preferred pages discoverable through normal internal links and, when useful, a sitemap. The sitemap and robots.txt operations guide covers the larger sitewide setup; this article focuses on deciding whether a crawl rule is appropriate.
How Do You Check a Rule Before and After Publishing?
- Open the exact host's
/robots.txtURL. Read the rule group that applies to the crawler you care about. - Choose representative URLs: one intended to be blocked and one important page that must stay crawlable. Compare the full paths against the rule. Avoid publishing a broad
Disallow: /by accident. - After the file is live, use Search Console's URL Inspection on important pages to check the reported crawl permission, and run its live test where appropriate. A crawlable result does not guarantee indexing or ranking.
Record why each restriction exists, and review it when the site structure changes.
Frequently Asked Questions
Is robots.txt Still Relevant?
Yes, when a site needs to manage crawler access to specific URLs. A small, normally linked site may not need any special restrictions.
Does robots.txt Prevent Indexing?
No, not reliably for webpages. It stops compliant crawlers from fetching a matching URL, but Google may still list the URL if it learns about it elsewhere. Use crawlable noindex for an index-exclusion decision.
How Can I View a Website's robots.txt File?
Open /robots.txt at the root of the exact protocol and host you want to check, such as https://example.com/robots.txt. A subdomain or other host can have a different file.
When Should I Use noindex Instead?
Use noindex when a publicly accessible page should not appear in Google Search. Let Google crawl that page so it can read the instruction. For private content, require access control rather than either directive.