Robots.txt Generator
NewBuild robots.txt files with a form UI, common presets including Block AI Bots, and plain-English explanations.
User-agent: * Allow: /
Related Tools
Embed this tool
Add this tool to your website or blog with a single iframe snippet.
<iframe src="https://dukotools.com/tools/robots-txt-generator?embed=1" width="100%" height="600" frameborder="0" allow="clipboard-write" loading="lazy" title="robots-txt-generator tool"></iframe>
About Robots.txt Generator
Robots.txt Generator creates valid robots.txt files through a form UI without requiring you to remember directive syntax. The user agents section lets you add multiple rules: select a bot from a dropdown of common crawlers including Googlebot, Bingbot, GPTBot, CCBot, and DuckDuckBot, or type any custom user agent. Each rule has separate allow paths and disallow paths inputs. The presets panel provides one-click solutions: Block AI Training Bots adds Disallow for GPTBot, CCBot, anthropic-ai, and other known AI training crawlers; Block All Bots; Allow All Bots; WordPress Standard; and Block Everything Except Homepage. The plain-English explanation panel translates every generated directive into readable sentences.
Key Features
Form-Based UI
Build robots.txt through a structured form without memorising directive syntax.
Block AI Training Bots Preset
One-click preset adds Disallow for GPTBot, CCBot, anthropic-ai, and all known AI training crawlers.
Common Bot Dropdown
Select Googlebot, Bingbot, GPTBot, CCBot, DuckDuckBot or type a custom user agent.
Multiple User Agent Rules
Add as many user agent blocks as needed with individual allow and disallow paths.
Plain-English Explanations
Each generated directive is explained in plain readable English beside the output.
Sitemap URL Directive
Adds a Sitemap directive pointing crawlers to your sitemap.xml.
Download as robots.txt
Downloads the file with the correct filename for placing in your site root.
How to Use
- 1
Choose a preset (optional)
Click a preset card for instant common configurations like Block AI Bots or WordPress Standard.
- 2
Add user agent rules
Select or type a user agent and enter allow and disallow paths.
- 3
Add your sitemap URL
Enter your sitemap.xml URL in the Sitemap input.
- 4
Read the explanations
Review the plain-English explanation of each generated directive.
- 5
Download or copy
Download the file as robots.txt or copy the content.
Real-World Use Cases
Blocking AI Data Scrapers
A blog owner uses the Block AI Training Bots preset to add Disallow rules for all known AI training crawlers to protect their content.
E-commerce Site Configuration
An e-commerce developer uses the form to allow product pages while disallowing cart, checkout, admin, and internal search pages.
Media Site Configuration
A news publisher uses the WordPress Standard preset and adds custom rules to block access to author archive pages that duplicate content.
Frequently Asked Questions
Related Free Tools
Explore these tools that work great alongside Robots.txt Generator: