Free Robots.txt Checker
Test which robots.txt rule applies to a URL and crawler.
See which rule actually matches
Paste the root robots.txt for your target URL. Test a crawler token and see the matching rule and line number.
Frequently Asked Questions
Which rules does this checker use?
It uses the most specific matching user-agent group and URL rule. Matching Allow rules win ties. Wildcards and end-of-path markers are supported.
Does Disallow remove a page from Google?
Not necessarily. Robots.txt controls crawling, not removal from the index.
