Rule note
Crawl-delay is only respected by some non-Google agents — Google does not honour the Crawl-delay directive, so this generator only writes it for groups that name a non-Google agent. Enter paths one per line in the Allow/Disallow lists; blank lines separate each user-agent group, exactly as the standard expects.
Rules configuration
Only written for non-Google agents — Google ignores this directive.