Free Robots.txt Generator – Create a Robots.txt File Online

A robots.txt file tells search engines which parts of your website they can and cannot crawl. A small mistake in it can hide your best pages from Google, so it pays to get it right. Our free Robots.txt Generator lets you build a clean, error-free file without memorising any syntax. Pick a preset for WordPress or WooCommerce, add your own Allow and Disallow rules, block AI and SEO crawlers, and include your sitemap URL. You see the result live, and you can copy it or download it as a ready-to-upload robots.txt file.

Quick preset

Rules for all crawlers (User-agent: *)

Disallow paths (one per line)

Allow paths (one per line, optional)

Crawl-delay (seconds, optional)

Block specific bots completely

AI and data crawlers

SEO tool crawlers

Custom rules for a specific bot

Sitemaps

Add a comment header with today's date

Your robots.txt

HOW TO USE

Choose a quick preset such as WordPress, WordPress + WooCommerce, Block AI bots, Allow all, or Block all.
Add the folders or pages you want to keep crawlers out of in the Disallow box, one path per line (for example /wp-admin/).
Use the Allow box for exceptions, such as /wp-admin/admin-ajax.php.
Optionally set a Crawl-delay. Note that Google ignores this setting.
Tick any AI or SEO bots you want to block completely, such as GPTBot, ClaudeBot, CCBot, AhrefsBot or SemrushBot.
Need rules for one specific bot? Click “Add bot rule” and enter its name and paths.
Paste your sitemap URL, for example https://yourwebsite.com/sitemap.xml.
Check the preview and any warnings, then click Copy or Download robots.txt.
Upload the file to the root of your site so it opens at yourwebsite.com/robots.txt. You can also paste it into your SEO plugin’s robots.txt editor.

FAQs
Q1. What is a robots.txt file?
It is a plain text file in your site’s root folder that gives crawling instructions to search engine bots.

Q2. Does robots.txt remove a page from Google?
No. It only stops crawling. To keep a page out of search results, use a noindex tag instead.

Q3. Where do I upload robots.txt?
In the root directory of your domain, so it loads at yourdomain.com/robots.txt.

Q4. Should I block AI bots like GPTBot?
It is optional. If you don’t want AI companies to crawl your content for training, tick those bots in the generator.

Q5. Can I block CSS and JavaScript files?
It is not recommended. Google needs them to render your pages, and blocking them can hurt your rankings.

Q6. Is this tool free?
Yes, completely free, with no signup, and everything runs in your browser.