robots.txt Generator

Generate a standards-compliant robots.txt for your site. Build crawl-rule groups per user-agent (allow/disallow path patterns, crawl-delay), reference your sitemap(s), and set a Host directive. Includes presets for allow all, block all, and blocking common AI training crawlers (GPTBot, ClaudeBot, Google-Extended, and more). Pairs with the Sitemap Generator. Everything runs in your browser; nothing is uploaded.

Sitemap URLs (one per line)

Host directive (optional)

robots.txt

Each group's User-agent field accepts one agent per line. Path patterns must begin with /. An empty Disallow allows everything for that group. In robots.txt the most specific matching group wins, so list dedicated bot groups before a catch-all * group.