robots.txt Generator

What is robots.txt Generator?

robots.txt Generator is a free in-browser tool that writes the robots.txt file web crawlers read at the root of your site to decide what they may fetch. You set the User-agent (default *), list one Disallow path per line and any Allow exceptions, optionally add a Crawl-delay and a Sitemap URL, and the tool turns it all into the correct directives. It also includes a checkbox preset that blocks well-known AI training crawlers — GPTBot, ClaudeBot, CCBot, and Google-Extended — by appending a separate group for each. Site owners, SEO specialists, and developers use it when launching a site, hiding admin or staging paths from search engines, or opting out of AI scraping. Copy the result into a file named robots.txt at your domain root, or use the Download button.

How to use robots.txt Generator

1. Enter the user-agent the rules apply to, or leave it as * for all crawlers.
2. Type the paths to block in the Disallow box, one per line (for example /admin/ or /private/).
3. Add any Allow exceptions, one per line, to re-permit sub-paths inside a blocked folder.
4. Optionally set a crawl-delay in seconds and paste your full sitemap URL.
5. Tick the AI-crawler checkboxes (GPTBot, ClaudeBot, CCBot, Google-Extended) to block them from scraping your content.
6. Copy the generated robots.txt or click Download, then place the file at the root of your site.

Examples

User-agent: *
Disallow: /admin/
Disallow: /staging/
Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /admin/
Disallow: /staging/

Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /downloads/
Allow: /downloads/public/

User-agent: *
Disallow: /downloads/
Allow: /downloads/public/

Block GPTBot, ClaudeBot, CCBot, Google-Extended

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Frequently asked questions

What does robots.txt actually do?

It is a plain-text file at your site root that tells compliant crawlers which paths they may or may not request, using Allow and Disallow rules grouped by User-agent. It is a request, not an enforcement: well-behaved bots like Googlebot honor it, but it does not block access for tools that ignore it, so never use it to hide truly sensitive data.

How do the AI-crawler block presets work?

Each checkbox appends a dedicated group such as 'User-agent: GPTBot' followed by 'Disallow: /', which asks that crawler not to fetch any page. GPTBot is OpenAI's crawler, ClaudeBot is Anthropic's, CCBot is Common Crawl, and Google-Extended controls use of your content for Google's AI models. Ticking a box adds its group; unticking removes it.

What is the difference between Allow and Disallow?

Disallow lists path prefixes a crawler should not request, while Allow re-permits a more specific sub-path inside a disallowed folder. For example, Disallow: /files/ with Allow: /files/public/ blocks the folder but keeps the public sub-folder crawlable. Enter one path per line in each box.

Where do I put the generated file?

Save it as a file literally named robots.txt and upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers only look there; a robots.txt in a sub-folder is ignored. Use the Download button to save it directly.

Is anything I type sent to a server?

No. The robots.txt is generated entirely in your browser with JavaScript. Your paths, user-agent, sitemap URL, and choices are never uploaded, stored, or transmitted, so it is safe to draft rules for private or pre-launch sites.