Free seo tool · by Natwar Rathore Creative World
robots.txt Generator
Build a clean robots.txt with sitemap links, sensible presets and plain-English warnings.
Runs in your browser · nothing uploaded
How it works
How to use the robots.txt generator
A robots.txt file sits at the root of your site (your-site.com/robots.txt) and tells crawlers which areas they may visit. It's useful for keeping bots out of admin pages, carts and search-result pages, and for pointing them to your XML sitemap.
Start from a preset, add one path per line, and the generator writes the file with the correct syntax. You can also block common AI training crawlers by user-agent if you prefer your content not to be used that way; well-behaved bots follow these rules.
Two important cautions: robots.txt is a request, not a security lock, so never rely on it to hide private files; and blocking a page does not remove it from Google. To keep a page out of search results, allow crawling and add a noindex robots meta tag instead. Never block CSS or JavaScript that your pages need.
- Pick a preset or start from “Allow everything”.
- Add disallow and allow paths, one per line.
- Add your sitemap URL.
- Download robots.txt and upload it to your site's root folder.
FAQ
Questions about this tool
Where do I put robots.txt?
In the root of your domain so it opens at https://your-site.com/robots.txt.
Will Disallow remove a page from Google?
No. Use a noindex meta tag for that. Disallow only stops crawling, and a blocked URL can still appear if other sites link to it.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay; some other search engines respect it.
Should I block AI crawlers?
It's your choice. Blocking them doesn't affect normal Google Search rankings.
Hisar → India → Worldwide · Hindi & English
Want your website's SEO done? WhatsApp Natwar
Free tools show you what to fix. If you'd rather have it handled, send your website link for a clear plan and quote before anything starts. No ranking guarantees, just honest work.
Keep going