Free Robots.txt Checker
Test which robots.txt rule applies to a URL and crawler.
See which rule actually matches
Paste the root robots.txt for your target URL. Test a crawler token and see the matching rule and line number.
This checks pasted rules only. It does not test server permissions, firewalls or indexing, and it does not fetch robots.txt.
Frequently Asked Questions
Which rules does this checker use?
It uses the most specific matching user-agent group and URL rule. Matching Allow rules win ties. Wildcards and end-of-path markers are supported.
Does Disallow remove a page from Google?
Not necessarily. Robots.txt controls crawling, not removal from the index.
Try another free SEO tool
Choose the next task you want to work on.
Website SEO CheckerCheck a public page for titles, headings, indexing signals, links and structured data.
Schema Markup GeneratorCreate JSON-LD for an article, local business or product.Schema Markup ValidatorCheck JSON-LD syntax and common fields before publishing.XML Sitemap GeneratorTurn a list of page URLs into a downloadable XML sitemap.