Robots.txt generator
Tell search engines and other bots which parts of your site they may visit.
Sealed on this device
Runs in your browser. Nothing is uploaded.
Sealed
Rules
One path per line, starting with /. Use * as a wildcard and $ for the end of an address.
Optional. Bing and some other bots follow it; Google ignores it.
Specific crawlers
One full address per line, for example https://example.com/sitemap.xml.
Your robots.txt
Things to check
Test an address
Leave empty to block the whole site for this crawler.
How it works
-
Choose the general rule
Allow everything, block everything, or list the paths crawlers should skip. Presets help you start.
-
Add AI bots, crawlers and sitemaps
Block AI training crawlers in one click, add rules for specific bots, and list your sitemaps.
-
Test and publish
Test a few addresses, then download the file and put it at the root of your site: example.com/robots.txt.
Frequently asked questions
Where do I put the robots.txt file?
At the root of your domain, so it’s reachable at https://your-site.com/robots.txt. Each subdomain needs its own file. Crawlers don’t look for it anywhere else.
Does robots.txt hide a page from Google?
No. It asks crawlers not to visit a page, but the page can still appear in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag, or protect it with a password.
How do I block AI crawlers like GPTBot?
Tick “Block AI training crawlers”. The file then tells GPTBot, ClaudeBot, Google-Extended and other training bots to stay away from your whole site. Well-behaved bots respect it, but robots.txt is a request, not a lock.
What happens when Allow and Disallow rules conflict?
Google and most modern crawlers apply the most specific rule, the one with the longest matching path. When two rules are equally long, Allow wins. The tester on this page follows the same logic.
What do * and $ mean in a path?
The asterisk matches any sequence of characters, so /*.pdf matches every PDF. The dollar sign marks the end of the address, so /*.pdf$ matches addresses that end exactly with .pdf.
Should I add my sitemap to robots.txt?
Yes, it’s a simple way to help every search engine find it. Use the full address, including https://. You can list several sitemaps.