robots.txt generator
Choose a preset or write rules for each crawler, add your sitemap links, and copy a robots.txt that follows RFC 9309. Paths are checked as you type.
Runs in your browser: nothing you enter is sent anywhere.
How to use it
- Pick a preset: Allow all, Block all, or Typical site, then edit it.
- For each group, name the robots in User-agent and list the paths to disallow or allow.
- Add your sitemap URLs, then copy the text into a file named
robots.txt.
A robots.txt is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules for those robots. Crawl-delay is not part of the standard. Sitemap lines are independent of groups and must hold full URLs.
User-agent: *
Allow: /admin/help.html
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
- Disallow and Allow paths must start with
/. An empty Disallow is created for you when a group has no rules.
- Robot names may contain letters, digits,
- and _, or be *.
- Crawl-delay must be a number of seconds.
- Sitemap entries must be full http or https URLs.
- Lines that fail a check are listed above the output and are left out of the file.
Tips
Paths are matched from the start of the URL path, so /shop also blocks /shop-sale. End a folder with / to match only that folder. Do not block CSS and JavaScript files that your pages need to render, because search engines use them to understand the page. Parsers must read at least the first 500 KiB of the file, so keep it short.
Frequently asked questions
What does an empty Disallow mean?
Disallow: with nothing after it means no path is blocked, so the group allows everything. To block the whole site, use Disallow: /. This tool writes an empty Disallow when a group has no other rules.
Does robots.txt hide pages from Google?
No. It asks crawlers not to fetch a URL, but a blocked URL can still appear in results if other pages link to it. To keep a page out of search, let it be crawled and add a noindex meta tag or header. Never rely on robots.txt for privacy; the file itself is public.
How do Allow and Disallow rules interact?
A crawler follows the most specific rule, which is the longest matching path. If an Allow and a Disallow match equally well, Google uses the less restrictive one. So Disallow /admin/ with Allow /admin/help.html blocks the folder but still permits that page.
Do crawlers combine several groups?
No. A crawler obeys the one group that names it most specifically and ignores the others, including the * group. If you add a Googlebot group, repeat any shared rules in it.
Does Google respect Crawl-delay?
No, Google ignores Crawl-delay. Bing reads it, and other crawlers may. To slow Google down you have to rely on server response codes or the tools in Search Console.