Free tools SEO

Robots.txt tester

Check whether a crawler is allowed to fetch a page under your robots.txt rules.

Fetch public robots.txt (optional)

Test pasted or fetched rules

Simulation runs entirely in this browser. Maximum 64 KiB / 2,000 lines. No page is fetched when testing a path.

This bounded simulator implements longest path match, Allow ties, wildcard and end anchors, and merged best-matching groups. It does not certify exact Googlebot behavior. Robots.txt is not authentication and does not guarantee deindexing.

Free, no sign-up. Results stay in this tab. What tools measure and save

What it checks

  • Fetch a live robots.txt, or paste your own rules
  • Test any path for any user agent (Googlebot, GPTBot…)
  • Shows the exact rule that decides the result
  • Allow vs Disallow precedence and wildcards
  • Warnings for common mistakes

How it works

  1. 1

    Optionally fetch a website's public robots.txt, or paste your own rules.

  2. 2

    Choose a user agent and enter a path beginning with /.

  3. 3

    Inspect the matching directive and warnings; robots is not authorization or deindexing.

Example: Test /private/help against Disallow: /private/ and Allow: /private/help$.

About the robots.txt tester

robots.txt tells crawlers which parts of your site they may fetch. One wrong line can hide your whole site from search engines or AI crawlers.

Note that robots.txt is a request, not a lock: it doesn't make pages private, and blocking a page doesn't remove it from search results by itself.

Checked your site? Launch it.

Launch on ToastLaunch for free: a permanent product page, a week on the board and a receipt of what the launch earned. Fill with AI does the form for you.

Launch your site free →

Questions

How do I block AI crawlers?

Add a group for each crawler, for example "User-agent: GPTBot" followed by "Disallow: /". Test it here before publishing.

Will Disallow remove a page from Google?

Not by itself. To keep a page out of results, allow crawling and add a noindex tag instead.

Which rule wins when Allow and Disallow both match?

The most specific (longest) matching rule wins; the tester shows which one applied.

More free tools

Launch-day checklist →