Create rules for your crawlers
Choose whether to allow crawling, use custom path rules, or block crawling. Enter crawler tokens one per line and add an optional full sitemap URL. Custom paths must begin with a slash; use URL-encoded paths where spaces are needed.
Understand what robots.txt controls
Robots.txt requests crawler behavior. It is not a security mechanism and does not reliably remove a URL from search results. Do not list sensitive locations as a substitute for access controls. Disallowing all crawling can also prevent crawlers from seeing a page’s noindex directive.
Publish on the correct host
Review the file and upload it as robots.txt at the root of the relevant host. Rules for one subdomain do not automatically apply to another. This tool only generates text; it does not change ContentRyter’s file or your website.
Check Google’s robots.txt specification before applying complex rules. Allowed exceptions can override broader disallowed paths according to crawler matching rules.
Privacy and free usage
All processing happens in your browser. These utilities do not submit inputs to an AI service provider, save your entries, or use the AI allowance.