Browser-side utility

Robots.txt Generator

Build a standards-compliant robots.txt file with crawler rules, allow/disallow paths, and sitemap URL.

User-agent

Sitemap URL

Allow Paths (one per line)

Disallow Paths (one per line)

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

How It Works

Add user-agent rule blocks, specify disallow or allow paths, attach your sitemap URL, and download or copy robots.txt.

Does robots.txt prevent sensitive pages from being accessed?: No. Robots.txt only guides crawlers. Sensitive pages must be protected with server-side authentication.