robots.txt Tester

Paste your robots.txt and a URL — see whether the URL is allowed for a given user agent. Implements the standard precedence rules (most-specific match wins, longest matching path wins on tie). Useful when a page isn't indexing and you suspect a stray Disallow rule.

How to use the robots.txt Tester

Paste your robots.txt. Set the user agent you want to simulate (Googlebot, Bingbot, *, etc.) and the URL to test. The tool reports allowed/disallowed, which rule matched, and any Sitemap declarations.

How robots.txt rules are applied

The standard rule (RFC 9309, 2022) is: pick the user-agent section that most-specifically matches the requesting bot. Inside that section, the rule with the longest matching path wins. Allow and Disallow compete by path-length, not order. User-agent: * is the fallback when no more-specific UA matches.

Common mistakes: assuming the order of rules matters (it doesn't), forgetting that paths are prefix matches (Disallow: /admin blocks /admin and /admin-anything), using non-standard directives like Noindex in robots.txt (deprecated since 2019; use meta robots instead).