InstantTools.siteZero Latency Utilities
dev Tool
100% In-Memory (Zero Server Upload)

Robots.txt Generator

Build a validated robots.txt with user-agent groups, allow and disallow paths, crawl-delay and your sitemap URL — copy or download it.

Rule note

Crawl-delay is only respected by some non-Google agents — Google does not honour the Crawl-delay directive, so this generator only writes it for groups that name a non-Google agent. Enter paths one per line in the Allow/Disallow lists; blank lines separate each user-agent group, exactly as the standard expects.

Rules configuration

Group 1
All crawlers

Only written for non-Google agents — Google ignores this directive.

robots.txt output

About Robots.txt Generator

Assemble a robots.txt file without having to remember the syntax. Add one or more user-agent groups, give each its own allow and disallow rules in the order you want them written, set a crawl delay for the agents that support one, and point crawlers at your sitemap. The tool validates the sitemap URL, separates the groups correctly, and shows the finished file live so you can copy it or download robots.txt straight into your site's root folder.

Key Features & Privacy Guarantee

  • Zero server transmission: all computation happens instantly in your local browser runtime.
  • No sign-up, no subscriptions, and no usage quotas.
  • Optimized for maximum speed, accuracy, and lightweight performance.
  • Instant one-click clipboard copying.

Frequently Asked Questions

Common questions and helpful answers about this utility.

Neither treats Crawl-delay as a supported directive, and Google ignores it. It is still respected by a number of other crawlers, including some search bots and archiving services.