robots.txt Generator

Build a robots.txt with per-user-agent rules. Add Allow/Disallow lines for any path, declare sitemaps, set crawl-delay. Output is RFC 9309-compliant and works with all major crawlers.

How to use the robots.txt Generator

Tick the rules you want, optionally add custom lines. Generate produces the complete file. Place at the root of your domain at /robots.txt. Test the result with the robots.txt Tester.

Writing a correct robots.txt

A robots.txt file tells crawlers which parts of your site they may request. The syntax is small but easy to get subtly wrong — a stray Disallow: / can deindex an entire site, and rules apply per user-agent in ways that trip people up.

This generator builds a compliant file from checkboxes: allow everything by default, block common paths like /admin/ or /api/, add a sitemap declaration, and optionally block AI crawlers such as GPTBot. The output follows RFC 9309 and works with all major crawlers. Because robots.txt only controls crawling, not indexing, verify a specific page's true indexability with the noindex tester.

Common use cases

• New sites — produce a sensible robots.txt from scratch.
• Blocking admin areas — keep crawlers out of private paths.
• Sitemap discovery — declare your sitemap for search engines.
• AI opt-out — block GPTBot, Claude-User, and similar agents.
• Crawl control — add per-user-agent allow and disallow rules.