Robots.txt Generator
Build a robots.txt file with bot rules, AI blockers and sitemaps.
What is robots.txt?
Robots.txt is a small text file at the root of a website, for example https://example.com/robots.txt. Before a well behaved crawler visits your pages, it reads this file to see which parts of the site it is allowed to crawl. You can use it to keep crawlers out of admin areas, shopping carts and internal search pages, to point them to your XML sitemap, and to opt out of AI training crawlers such as GPTBot, CCBot or Google-Extended.
How to create a robots.txt file
- Pick a preset such as WordPress, WooCommerce or Block AI crawlers, or start from scratch.
- Choose the default for all robots: allow everything or block everything.
- List any folders or pages all robots should skip, and any exceptions to allow.
- Add rules for specific bots: block a bot entirely, allow it, or block certain paths.
- Enter your sitemap URL so search engines can find every page.
- Copy the result or download robots.txt and upload it to the root folder of your domain.
Options in this generator
- Default rule for all user agents with
User-agent: *. - Disallow and Allow paths, including wildcards such as
/*?add-to-cart=*. - Per bot rules for Googlebot, Bingbot, Yandex, Baidu, Applebot, GPTBot, ClaudeBot, PerplexityBot, SEO crawlers and custom bots.
- Crawl-delay, which Bing and Yandex respect but Google ignores.
- Sitemap lines with validation that each one is a full URL.
- Warnings for common mistakes such as paths without a leading slash.
Robots.txt for WordPress
WordPress creates a virtual robots.txt by default. A good basic file blocks /wp-admin/ but allows /wp-admin/admin-ajax.php, because many themes and plugins load content through it. Do not block /wp-content/ or /wp-includes/, since Google needs your CSS and JavaScript to render pages properly. If you use an SEO plugin, you can usually paste the generated rules into its robots.txt editor.
Important things to know
Robots.txt is a request, not a lock. Reputable crawlers follow it, but bad bots can ignore it, so never rely on it to hide private information. A disallowed page can still appear in search results without a description if other sites link to it; use a noindex meta tag or password protection to keep a page out of search entirely. Test your file in Google Search Console after uploading it, and be very careful with Disallow: /, which blocks the whole site.
Frequently asked questions
Where do I upload robots.txt?
Upload it to the root of your domain so it is available at https://yourdomain.com/robots.txt. Files in subfolders are ignored by crawlers.
Can I block ChatGPT and other AI crawlers?
Yes. Use the Block AI crawlers preset or add rules for bots such as GPTBot, ClaudeBot, CCBot, PerplexityBot and Google-Extended. Compliant AI crawlers respect these rules.
Does blocking a page in robots.txt remove it from Google?
Not always. It stops crawling, but the URL can still be listed if other pages link to it. Use a noindex tag on a crawlable page to remove it from results.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay. Bing, Yandex and several other crawlers respect it.