Robots.txt Generator
Build a robots.txt file for your website.
User-agent: * Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
Save this as robots.txt at the root of your domain.
How to use this tool
- 1Choose whether to allow or block all crawlers.
- 2Add any paths to disallow.
- 3Add your sitemap URL, then copy the generated file.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, e.g. https://example.com/robots.txt. It must be served exactly at that path.
Does robots.txt block pages from Google's index?
It stops crawling, not necessarily indexing. Use a noindex meta tag to keep pages out of search results.