Quikbench
Quikbench
Enter your website URL and get a best-practice robots.txt instantly. Every other option is optional — add only what you need, then download.
Block specific bots entirely. All bots are allowed by default.
Toggle standard paths you want to hide from crawlers.
No custom rules added.
# Enter your website URL and click # "Generate robots.txt" to start.
The robots.txt file lives at your domain root and tells crawlers which parts of your site they can and cannot access. Well-behaved bots — including Googlebot, Bingbot, and social media crawlers — follow these rules automatically. The file is public and advisory; it does not provide security, it just guides legitimate crawlers.
Upload the downloaded file so it is accessible at https://yourdomain.com/robots.txt. The file must be exactly at the root — subdirectory robots.txt files are not respected by most crawlers. After uploading, you can test it in Google Search Console under the URL Inspection tool.
The AI bot toggles (GPTBot, CCBot, Google Extended, ClaudeBot) let you opt out of having your content used for AI training datasets — a growing concern for content publishers. These are well-supported by the major AI companies.
The single most common robots.txt mistake is a stray "Disallow: /" left over from a staging environment, which tells every well-behaved crawler to skip the entire site — including Google. Always double-check the generated file against your intended rules before uploading it, and verify it in Google Search Console's robots.txt testing tool once live.
Blocking a page in robots.txt stops crawlers from fetching it, but doesn't guarantee it won't be indexed — if other sites link to a blocked URL, Google can still list that URL in search results without a description, since it can see the URL exists even though it can't crawl the page's content. To truly prevent a page from appearing in search results, use a noindex meta tag on the page itself instead of, or in addition to, robots.txt.
Most major crawlers (Google, Bing, and others) respect robots.txt as a convention, but it's not legally enforced — a poorly-behaved bot can simply ignore it.
At the root of your domain (e.g. yoursite.com/robots.txt) — placing it in a subfolder means crawlers won't find it.
📖 Related guide: How robots.txt Works, and Common Mistakes That Hurt SEO