Robots.txt Generator & Tester

Create a correct robots.txt in a minute — with presets for your platform, one-click blocking of AI crawlers, and your sitemap — then test exactly which pages each crawler can visit.

  • Free, no sign-up — runs in your browser
  • Block GPTBot, ClaudeBot, Google-Extended and other AI crawlers in one click
  • Test any page against any crawler, and catch mistakes
No sitemap yet? Make one free →
Block AI crawlersThese don't affect Google or Bing search results. ·
More options

Frequently asked questions

What is robots.txt?

A small text file at the root of your website (yoursite.com/robots.txt) that tells search engines and other crawlers which pages they may visit. It's read before they crawl anything else.

Will blocking AI crawlers hurt my Google ranking?

No. GPTBot, ClaudeBot, CCBot, Google-Extended and the other training crawlers are separate from Googlebot and Bingbot, so blocking them doesn't change your search results. The exceptions are AI search crawlers such as OAI-SearchBot and PerplexityBot — blocking those can keep your pages out of those AI tools' answers, which is why “Tick all training bots” leaves them unticked.

Does robots.txt hide a page from Google?

Not reliably. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag instead.

Where do I put the file?

In the main folder of your website, so it opens at yoursite.com/robots.txt. On WordPress, many SEO plugins (Yoast, Rank Math) let you edit it in the dashboard. On Shopify, edit the robots.txt.liquid theme template.

What does the tester do?

It checks a page against your rules the same way Google does: the most specific matching rule wins, and Allow wins a tie. It also points out common mistakes like typos, paths without a slash, and rules placed before any User-agent line.

Is anything uploaded?

No. The file is created and tested in your browser. Only when you click Load do we fetch that website's public robots.txt for you.

How to create a robots.txt

  1. Pick your platform to start from sensible rules — WordPress, WooCommerce, Shopify, or allow everything.
  2. Tick the AI crawlers you want to block, and add your sitemap address.
  3. Copy or download the file and upload it to your website's main folder.
  4. Test it — check your important pages are still allowed for Googlebot.

Careful: a single line, Disallow: / under User-agent: *, blocks your whole site from every search engine. Only use “Block everything” for staging or private sites.

Related tools