← All tools

robots.txt Rule Tester

See which rule decides each URL path for a crawler, and why that group and rule win.

Paste a file; nothing is fetched. Comments, blank lines and unrelated records do not end a group.
Matching is case-insensitive; the longest matching product token selects the groups.
One path per line, including any query. Absolute HTTP, HTTPS or FTP URLs are reduced to path + query.

Uses RFC 9309 longest-octet matching, * wildcards, terminal $, allow-on-tie, and Google-style most-specific User-Agent group selection with equal-specificity groups merged. /robots.txt is implicitly allowed. Crawler implementations can differ; these rules are crawl requests, not access control. No files, URLs or User-Agent strings are sent or stored.