Free tools SEO
Robots.txt tester
Check whether a crawler is allowed to fetch a page under your robots.txt rules.
This bounded simulator implements longest path match, Allow ties, wildcard and end anchors, and merged best-matching groups. It does not certify exact Googlebot behavior. Robots.txt is not authentication and does not guarantee deindexing.
Free, no sign-up. Results stay in this tab. What tools measure and save
What it checks
- Fetch a live robots.txt, or paste your own rules
- Test any path for any user agent (Googlebot, GPTBot…)
- Shows the exact rule that decides the result
- Allow vs Disallow precedence and wildcards
- Warnings for common mistakes
How it works
- 1
Optionally fetch a website's public robots.txt, or paste your own rules.
- 2
Choose a user agent and enter a path beginning with /.
- 3
Inspect the matching directive and warnings; robots is not authorization or deindexing.
Example: Test /private/help against Disallow: /private/ and Allow: /private/help$.
About the robots.txt tester
robots.txt tells crawlers which parts of your site they may fetch. One wrong line can hide your whole site from search engines or AI crawlers.
Note that robots.txt is a request, not a lock: it doesn't make pages private, and blocking a page doesn't remove it from search results by itself.
Checked your site? Launch it.
Launch on ToastLaunch for free: a permanent product page, a week on the board and a receipt of what the launch earned. Fill with AI does the form for you.
Questions
How do I block AI crawlers?
Add a group for each crawler, for example "User-agent: GPTBot" followed by "Disallow: /". Test it here before publishing.
Will Disallow remove a page from Google?
Not by itself. To keep a page out of results, allow crawling and add a noindex tag instead.
Which rule wins when Allow and Disallow both match?
The most specific (longest) matching rule wins; the tester shows which one applied.
More free tools
- Website SEO auditEnter a page to check its title, description, headings, links, images and indexability in seconds.
- GEO / AI readiness checkerSee whether AI assistants and search crawlers can read and understand your page.
- Meta tag generatorWrite your page title, description and social tags, preview them, and copy ready-to-paste HTML.
- Open Graph previewSee how your link will look when shared on X, Facebook and LinkedIn, before you post it.
- Sitemap checkerCheck your sitemap.xml for URLs, duplicates, dates and structure problems.
- Structured data validatorPaste JSON-LD structured data to check its syntax and schema.org structure.
- Keyword ideas generatorTurn one topic into dozens of search phrases and content ideas for your product.
- UTM link builderBuild tracked campaign links so you know which posts and channels bring visitors.
- QR code generatorCreate a QR code for any link or text, then download it as PNG or SVG.
- Website SEO comparisonRun the SEO audit on two pages side by side and see where each one is stronger.