Skip to content

ToolsRobots.txt Checker

Free Robots.txt Checker

Test which robots.txt rule applies to a URL and crawler.

See which rule actually matches

Paste the root robots.txt for your target URL. Test a crawler token and see the matching rule and line number.

Results appear here after a check.

Frequently Asked Questions

Which rules does this checker use?

It uses the most specific matching user-agent group and URL rule. Matching Allow rules win ties. Wildcards and end-of-path markers are supported.

Does Disallow remove a page from Google?

Not necessarily. Robots.txt controls crawling, not removal from the index.