Robots.txt
Generate a robots.txt file online for free. Set allow/disallow rules, sitemap URL, and crawl delay. Download or copy. No signup.
Runs entirely in your browser — files never uploaded
Preview
User-agent: * Disallow:
How to use Robots.txt
- 1. Add user-agent rules. Enter the bot name (or * for all bots) and specify which paths to allow or disallow.
- 2. Set global options. Add a sitemap URL and optional crawl delay that apply to every rule.
- 3. Use presets for speed. Click "Allow all", "Block all", or "Block AI bots" to start from a template.
- 4. Copy or download. Preview the result, then copy to clipboard or download as a robots.txt file to upload to your site root.
FAQ
Where do I put robots.txt?
Upload it to the root of your website so it's accessible at https://yoursite.com/robots.txt. Most web hosts let you drop it in the public or root folder.
Does robots.txt block pages from appearing in Google?
It tells crawlers not to fetch the page, but Google may still index the URL (without content) if other sites link to it. Use a noindex meta tag for stronger removal.
What is Crawl-delay?
It requests that crawlers wait a set number of seconds between requests. Google ignores it, but Bing and some others respect it.
Which AI bots does the "Block AI bots" preset cover?
GPTBot (OpenAI), ChatGPT-User, CCBot (Common Crawl, used by many AI training sets), Google-Extended (Gemini training), and anthropic-ai (Anthropic).
Related tools
- Base64 EncoderEncode text or files to Base64 or decode Base64 back online for free. Instant conversion, no signup, runs in your browser.
- JSON FormatterFormat and validate JSON online for free. Beautify with indentation, validate syntax, or minify. Runs in your browser, no signup.
- CSV ↔ JSONConvert CSV to JSON or JSON to CSV online for free. Live preview as you type or upload a file. No signup, runs in browser.
- Timestamp ConverterConvert Unix timestamps to dates and dates to timestamps online for free. Supports seconds, milliseconds, ISO 8601. No signup.
- JWT DecoderDecode JWT tokens online for free. Inspect header, payload, and claims. Verify expiration and signature algorithm. No signup, runs in browser.
- Regex TesterTest regular expressions online for free with live match highlighting, capture groups, and flag toggles. Supports JavaScript regex. No signup.