Robots.txt Generator & Validator

Create crawler rules, add sitemap directives, and validate robots.txt syntax locally before publishing the file at your website root.

Ready

Validation results

  • Generate or paste robots.txt content, then validate it.

Crawler rule generation

Build User-agent, Allow, Disallow, Sitemap, and optional Host directives without memorizing robots.txt formatting.

Syntax validation

Check common mistakes such as missing user-agents, relative sitemap URLs, malformed paths, duplicate rules, and unsupported directive names.

Private local processing

The generator and validator run in your browser. Robots.txt content is not uploaded to Convertr servers.

Frequently Asked Questions

Is my robots.txt file uploaded for validation?

No. Generation and validation run in your browser with JavaScript. The text you enter is not sent to Convertr servers.

Where should I place robots.txt?

Place robots.txt at the root of the host it applies to, such as https://example.com/robots.txt.

Does robots.txt remove pages from Google?

No. Robots.txt controls crawler access. Use noindex directives on the page or remove the page if you need deindexing.

Related developer tools

XML Sitemap Generator Create a valid sitemap.xml file from a URL list. HTTP Headers Checker Inspect redirects, status codes, and response headers. XML Formatter Format and validate XML files and sitemap snippets. URL Encoder & Decoder Encode URLs, query strings, and URL components. URL Slug Generator Create readable URL paths for pages and posts. Open Graph Checker Preview how pages and sitemaps appear on social media platforms. Cron Expression Generator Build and validate scheduled crawler or maintenance jobs.

Robots.txt Generator

Build and validate basic robots.txt groups, Allow and Disallow rules, crawl delays, and sitemap references.

Input
User-agent groups, path rules, and sitemap URLs.
Output
Copyable robots.txt text with structural diagnostics.
Real limitations
robots.txt is advisory, crawler support varies, rules are case-sensitive path prefixes, and it cannot remove an already indexed URL by itself.
Privacy and data handling
user-agent groups, path rules, and sitemap URLs is processed in this browser tab. Convertr does not receive or store it; closing or reloading the page clears the working state unless the tool explicitly offers a local download.

How to use this tool

  1. Provide user-agent groups, path rules, and sitemap URLs in the tool controls.
  2. Choose the settings that affect copyable robots.txt text with structural diagnostics, then run the tool.
  3. Review copyable robots.txt text with structural diagnostics before copying or downloading it.

Common failure modes

Invalid sitemap URLs, conflicting blank groups, unsupported directives, or a missing user agent produces warnings.

Useful example

Create a * user-agent group that disallows /admin/ and names the production sitemap URL.