robots.txt Tester

Paste your robots.txt and a URL — see whether the URL is allowed for a given user agent. Implements the standard precedence rules (most-specific match wins, longest matching path wins on tie). Useful when a page isn't indexing and you suspect a stray Disallow rule.

How to use the robots.txt Tester

Paste your robots.txt. Set the user agent you want to simulate (Googlebot, Bingbot, *, etc.) and the URL to test. The tool reports allowed/disallowed, which rule matched, and any Sitemap declarations.

How robots.txt rules are applied

The standard rule (RFC 9309, 2022) is: pick the user-agent section that most-specifically matches the requesting bot. Inside that section, the rule with the longest matching path wins. Allow and Disallow compete by path-length, not order. User-agent: * is the fallback when no more-specific UA matches.

Common mistakes: assuming the order of rules matters (it doesn't), forgetting that paths are prefix matches (Disallow: /admin blocks /admin and /admin-anything), using non-standard directives like Noindex in robots.txt (deprecated since 2019; use meta robots instead).

Frequently asked questions

How does it decide allow or disallow?
It applies the standard rules: the most specific user-agent group wins, and within it the longest matching path decides, with Allow breaking ties over Disallow.

Why is my page still indexed despite a Disallow?
Disallow blocks crawling, not indexing. A blocked URL can still appear in results from external links. Use a noindex meta tag to keep a page out of the index.

Does blocking in robots.txt remove a page from Google?
No. To remove a page, allow crawling and add a noindex directive, since Google must be able to crawl the page to see the noindex.
Embed this tool on your site

Free to embed, no attribution required (but appreciated). Paste this where you want the tool to appear: