DigitalWebs

Robots.txt Tester

Fetch a site’s robots.txt and test whether a URL is allowed for Googlebot or any other crawler.

Processed on our server. Your input is sent securely to our server for processing. Uploaded files are deleted automatically.

Requests are made from this website’s server. Only public web addresses can be checked; results are not stored.

How to use the Robots.txt Tester

  1. Enter the site or page address.
  2. Optionally change the path and user agent.
  3. See whether the path is allowed and which rule decided it.

Rules are evaluated the way RFC 9309 and Google specify: the most specific user-agent group applies, the longest matching path rule wins, Allow wins a tie, and * and $ wildcards are supported. robots.txt controls crawling, not indexing: to keep a page out of search results use a noindex tag instead.

Frequently asked questions

Does Disallow remove a page from Google?
No. A disallowed page can still be indexed if other pages link to it. Use noindex (and allow crawling so the tag can be seen).