robots.txt Generator crawler
Build a robots.txt from allow and disallow rules, crawl-delay, and a sitemap reference.
What this tool does
robots.txt is a text file at the site root that tells crawlers which paths they may fetch. It is the first thing a crawler requests when it visits.
The syntax is simple: User-agent names the crawler, Disallow lists paths to avoid, and Allow carves out exceptions. A Sitemap line at the end points crawlers at your page list so they discover content faster.
This tool offers presets for common setups and assembles a file from the paths you enter in the disallow and allow fields. It warns when a rule such as Disallow: / would block the entire site.
When to use it
Use it when launching a site and excluding admin or test paths from indexing, when blocking crawling on a staging server, when pointing crawlers at your sitemap, or when treating a particular crawler differently.
Input and output examples
Disallow /admin/ and /api/
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
The most typical arrangement.
Block-everything preset
User-agent: *
Disallow: /
For staging only. Deployed to production it removes the site from search.
Notes and limitations
robots.txt is not access control. It is a request that some crawlers ignore, and because the file is public it advertises the very paths you wanted to hide; protect sensitive routes with authentication. Disallow also does not guarantee exclusion from search results: a page linked from elsewhere can appear as a bare URL without its content being read. To exclude a page reliably use a noindex meta tag, and do not block crawling of that page, or the crawler will never see the tag. Deploying a staging block-everything file to production is a common accident, so verify before release.
Frequently asked questions
Can robots.txt hide a page reliably?
No. It only requests crawler behaviour and the file itself is public. Protect sensitive paths with authentication.
A disallowed page still appears in search.
It was discovered through links elsewhere. Use a noindex meta tag for reliable exclusion, and leave that page crawlable so the tag can be read.
Where does the file go?
At the domain root. It is only recognised at https://example.com/robots.txt.