Robots.txt Generator
Build a robots.txt file for your website.
User-agent: * Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
Save this as robots.txt at the root of your domain.
About Robots.txt Generator
Create a valid robots.txt file that tells search engine crawlers which parts of your site they may access. The robots.txt file lives at the root of your domain and controls crawler behavior, so a small mistake can accidentally hide your whole site from Google or expose pages you meant to keep private. This generator lets you set allow and disallow rules, target specific bots, and add your sitemap location, then copy a correctly formatted file. It is a time-saver for site owners and SEO professionals who want the syntax right the first time. Everything is generated locally in your browser with no signup.
How to use this tool
- 1Choose whether to allow or block all crawlers.
- 2Add any paths to disallow.
- 3Add your sitemap URL, then copy the generated file.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, e.g. https://example.com/robots.txt. It must be served exactly at that path.
Does robots.txt block pages from Google's index?
It stops crawling, not necessarily indexing. Use a noindex meta tag to keep pages out of search results.