Enter your website URL and get a best-practice robots.txt instantly. Every other option is optional — add only what you need, then download.
Block specific bots entirely. All bots are allowed by default.
Toggle standard paths you want to hide from crawlers.
No custom rules added.
# Enter your website URL and click # "Generate robots.txt" to start.
The robots.txt file lives at your domain root and tells crawlers which parts of your site they can and cannot access. Well-behaved bots — including Googlebot, Bingbot, and social media crawlers — follow these rules automatically. The file is public and advisory; it does not provide security, it just guides legitimate crawlers.
Upload the downloaded file so it is accessible at https://yourdomain.com/robots.txt. The file must be exactly at the root — subdirectory robots.txt files are not respected by most crawlers. After uploading, you can test it in Google Search Console under the URL Inspection tool.
The AI bot toggles (GPTBot, CCBot, Google Extended, ClaudeBot) let you opt out of having your content used for AI training datasets — a growing concern for content publishers. These are well-supported by the major AI companies.