Free Robots.txt File Generator
Directs bots (e.g. Bing, Yandex) to wait N seconds between request calls.
Sitemaps
Leave empty if you don't have one
Custom Path Rules (Global)
Block or allow specific directory paths or file patterns for all bots.
No custom path rules yet - add one above or use a quick-add shortcut below.
Search Robots
Block AI Crawlers
Disallow GPTBot, ClaudeBot, CCBot, Google-Extended, and other AI training/scraping bots.
Bot-Specific Directives (AI & Engines)
Configure distinct rules for specific search engines or AI web scrapers.
No bot-specific rules yet - add one to target a specific search engine or AI crawler by name.
Upload the generated file to /robots.txt in your site's root directory.
Your robots.txt file is formatted correctly for Googlebot & web standard scrapers.
Select a Preset
How to Use the Robots.txt Generator
Create a robots.txt to control crawling and indexing for your site - follow these basic steps.
1. Set Defaults or Apply a Template
Choose the global Allow/Disallow rule, crawl-delay, and sitemaps — or start from a preset like WordPress or E-Commerce.
2. Add Path Rules & Overrides
Add custom Allow/Disallow path rules, block AI crawlers, or override behavior for specific search robots.
3. Copy or Download
Copy the generated robots.txt or download it and upload it to your site root at /robots.txt.
Frequently Asked Questions
A robots.txt file is a plain text file placed in your website's root directory that tells search engine crawlers which pages or files they can or cannot request from your site. It helps manage crawler traffic, prevent server overload, and keep non-public pages from being crawled.
The robots.txt file must be placed in the top-level root directory of your domain (e.g., https://yourdomain.com/robots.txt. Search engine bots will only check for the file in this exact location.
No. A robots.txt file prevents bots from crawling a page, but if other sites link to that blocked URL, Google may still index the page title without crawling its content. To completely keep a page out of search results, use a noindex tag or password protection instead.
Disallow instructs crawlers not to visit a specific URL or path, while Allow explicitly permits access to a page or subfolder located inside an otherwise disallowed directory.
Still stuck? Contact us.
Explore More Tools
Discover more free tools to help you build, test, and optimize your site.
